Translation and multimodality : beyond words 9781138324428, 1138324426, 9781138324435, 1138324434

Table of contents :
Contents
Acknowledgements
List of figures
List of contributors
Introduction Monica Boria and Marcus Tomalin
1. Transposing meaning: translation in a multi-modal semiotic landscape Gunther Kress
2. A theoretical framework for a multi-modal conception of translation Klaus Kaindl
3. Meaning-(re)making in a world of untranslated signs: towards a research agenda on multimodality, culture, and translation Elisabetta Adami and Sara Ramos Pinto
4. From the "cinema of attractions" to danmu: a multimodal-theory analysis of changing subtitling aesthetics across media cultures' Luis Perez-Gonzalez
5. Translating "I": Dante, literariness and the inherent multi-modality of language Matthew Reynolds
6. The multi-modal dimensions of literature in translation Marcus Tomalin
7. Translations between music and dance: analysing the choreo-musical gestural interplayin twentieth- and twenty-first-century dance works Helen Julia Minors
8. Writing drawingly: a case study of multimodal translation between drawing and writing Tamarin Norwood
Beyond Words: concluding remarks Angeles Carreres and Maria Noriega-SanchezIndex

Citation preview

TRANSLATION AND MULTIMODALITY

Translation and Multimodality: Beyond Words is one of the first books to explore how translation needs to be redefined and reconfigured in contexts where multiple modes of communication, such as writing, images, gesture, and music, occur simultaneously. Bringing together world-­leading experts in translation theory and multimodality, each chapter explores important interconnections among these related, yet distinct, disciplines. As communication becomes ever more multimodal, the need to consider translation in multimodal contexts is increasingly vital. The various forms of meaning-­making that have become prominent in the twenty-­first century are already destabilising certain time-­honoured translation-­theoretic paradigms, causing breaking old definitions and assumptions to appear inadequate. This ground-­ volume explores these important issues in relation to multimodal translation with examples from literature, dance, music, TV, film, and the visual arts. Encouraging a greater convergence between these two significant disciplines, this text is essential for advanced students and researchers in Translation Studies, Linguistics, and Communication Studies. Monica Boria is Affiliated Lecturer in the Department of Italian at the University of Cambridge and a translator. Ángeles Carreres is Senior Language Teaching Officer in the Department of Spanish and Portuguese at the University of Cambridge. María Noriega-­Sánchez is Senior Language Teaching Officer in Spanish at the University of Cambridge, and a Fellow in Modern Languages at Sidney Sussex College. Marcus Tomalin is Fellow at Trinity Hall, Cambridge, and a Senior Research Associate in the Cambridge Machine Intelligence Laboratory.

TRANSLATION AND MULTIMODALITY Beyond Words

Edited by Monica Boria, Ángeles Carreres, María Noriega-­Sánchez, and Marcus Tomalin

First published 2020 by Routledge 2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN and by Routledge 52 Vanderbilt Avenue, New York, NY 10017 Routledge is an imprint of the Taylor & Francis Group, an informa business © 2020 selection and editorial matter, Monica Boria, Ángeles Carreres, María Noriega-­Sánchez, and Marcus Tomalin; individual chapters, the contributors. The right of Monica Boria, Ángeles Carreres, María Noriega-­Sánchez, and Marcus Tomalin to be identified as the authors of the editorial material, and of the authors for their individual chapters, has been asserted in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988. All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. Trademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. British Library Cataloguing-­in-­Publication Data A catalogue record for this book is available from the British Library Library of Congress Cataloging-­in-­Publication Data A catalog record has been requested for this book ISBN: 978-­1-­138-­32442-­8 (hbk) ISBN: 978-­1-­138-­32443-­5 (pbk) ISBN: 978-­0-­429-­34155-­7 (ebk) Typeset in Bembo by Apex CoVantage, LLC Visit www.routledge.com/9781138324435

This book is dedicated to our good friend and former colleague, Nicole Robertson, without whom it would never have existed.

CONTENTS

List of figures ix List of contributors x Acknowledgementsxii Introduction1 Monica Boria and Marcus Tomalin 1 Transposing meaning: translation in a multimodal semiotic landscape24 Gunther Kress 2 A theoretical framework for a multimodal conception of translation49 Klaus Kaindl 3 Meaning-­(re)making in a world of untranslated signs: towards a research agenda on multimodality, culture, and translation Elisabetta Adami and Sara Ramos Pinto 4 From the “cinema of attractions” to danmu: a multimodal-­ theory analysis of changing subtitling aesthetics across media cultures Luis Pérez-­González

71

94

viii Contents

5 Translating “I”: Dante, literariness, and the inherent multimodality of language Matthew Reynolds 6 The multimodal dimensions of literature in translation Marcus Tomalin 7 Translations between music and dance: analysing the choreomusical gestural interplay in twentieth-­and twenty-­ first-­century dance works Helen Julia Minors

117 134

158

8 Writing drawingly: a case study of multimodal translation between drawing and writing Tamarin Norwood

179



198

Beyond words: concluding remarks Ángeles Carreres and María Noriega-­Sánchez

Index

204

FIGURES

0.1 0.2 1.1 1.2 1.3 1.4a 1.4b 1.5

The squiggle flourish 4 Tramonto 15 The translation game 25 Drawing by a three-­year-­old: “this is a car” 34 Plant cell 37 Medical student touches patient 40 Surgeon touches patient 40 “Fixing” the ephemeral: documenting meaning in a game of “chase” 43 3.1 Bleeding nose used in anime to signify sexual arousal 73 3.2–3.4 Amy, Penny, and Bernadette go wedding dress shopping 81 4.1 Screenshot of video material featuring danmu107 6.1 Le  Dromadaire 143 6.2 New image 149 6.3 New text 149 7.1 Mapping music-­dance interdependency read through the 164 translation of languages, senses, and cultures 7.2 Bars 79–82 167 7.3a Bar 1 167 7.3b Bar 94 167

CONTRIBUTORS

Elisabetta Adami is University Academic Fellow in Multimodal Communication

at the University of Leeds. She specialises in social semiotic multimodal analysis, with a recent focus on inter-­, trans-­, and cross-­cultural contexts. Her publications include works on sign-­making practices in face-­to-­face interaction, and in digital environments. She is editor of Visual Communication. Monica Boria is Affiliated Lecturer in the Department of Italian at the University

of Cambridge and a translator. Her research interests are in contemporary Italian cultural studies and humour studies. With Linda Risso she co-­edited Laboratorio di nuova ricerca. Investigating Gender, Translation & Culture in Italian Studies (Troubador, 2007). She has translated the first Italian edition of Dorothy Edwards’s short stories Ammutinamento e altri racconti (ETS, 2019). Ángeles Carreres is Senior Language Teaching Officer in the Department of Span-

ish and Portuguese at the University of Cambridge. She is the author of Cruzando límites: la retórica de la traducción en Jacques Derrida (Peter Lang, 2005) and, with María Noriega-­Sánchez and Carme Calduch, of Mundos en palabras: Learning Advanced Spanish Through Translation (Routledge, 2018). Klaus Kaindl is Professor of Translation Studies at the University of Vienna, Austria.

His research interests range from the translation of multimodal/multimedial texts (e.g., opera, comics, popular music), to translation theory, translation sociology, and fictional representations of translators and interpreters. Gunther Kress was Professor of Semiotics and Education at the UCL Institute of Education, University of London. He was interested in communication and

Contributors  xi

meaning (-­making) in contemporary environments, and was instrumental in developing a social semiotic theory of meaning-­making and (multimodal) communication. Helen Julia Minors is Associate Professor of Music and Course Leader of degrees in Music and Creative Music Technology at Kingston University, London. She was co-­investigator for the AHRC project “Translating Music,” and is current co-­lead of the award-­winning project “Taking Race Live.” María Noriega-­Sánchez is Senior Language Teaching Officer in Spanish at the

University of Cambridge and a Fellow in Modern Languages at Sidney Sussex College. She is particularly interested in translation pedagogy and has recently co-­ authored with Ángeles Carreres and Carme Calduch the book Mundos en palabras: Learning Advanced Spanish through Translation (Routledge, 2018). Tamarin Norwood is an artist and writer. She was recently artist/writer-­ in-­ residence at Spike Island Bristol and was part of Hubbub, the Wellcome Collection’s inaugural interdisciplinary residency. She gained her doctorate at the Ruskin School of Art, University of Oxford, in 2018. Luis Pérez-­González is Professor of Translation Studies and Co-­director of the Centre for Translation and Intercultural Studies at the University of Manchester. He is author of Audiovisual Translation:Theories, Methods and Issues (Routledge 2014) and editor of The Routledge Handbook of Audiovisual Translation (2018). Sara Ramos Pinto is Lecturer in Translation Studies at the University of Leeds. She

has an enduring interest in the translation of dialects, but her overall interest lies in audiovisual and theatre translation, and the challenges that multimodal products bring to translation theory and practice. Recently, she has approached these issues from the perspective of experimental reception studies. Matthew Reynolds is Professor of English and Comparative Criticism at Oxford, where he chairs the Oxford Comparative Criticism and Translation Research Centre (OCCT). He is the author of Translation: A Very Short Introduction (2016), Likenesses:Translation, Illustration, Interpretation (2013), and The Poetry of Translation: From Chaucer & Petrarch to Homer & Logue (2011). Marcus Tomalin is a Fellow at Trinity Hall, Cambridge, and Senior Research Associate

in the Cambridge Machine Intelligence Laboratory. He is interested in the relationships between natural language, mathematics, and philosophy, and his many publications include Linguistics and the Formal Sciences (CUP, 2006), Romanticism and Linguistic Theory: William Hazlitt, Language, and Literature (Palgrave, 2009), “And he knew our language”: Missionary Linguistics on the Pacific Northwest Coast (John Benjamins, 2011), and The French Language and British Literature, 1756–1830 (Routledge, 2016).

ACKNOWLEDGEMENTS

This book emerged from the workshops and panel discussions organised by the research group Cambridge Conversations in Translation (CCiT) that was based in the Centre for Research in the Arts, Social Sciences, and Humanities (CRASSH) at the University of Cambridge from 2015–2018. We are especially grateful to the support and encouragement we received from Simon Goldhill and Steve Conner (successive Directors of CRASSH), Tim Lewens and Jan-­Melissa Schramm (successive Deputy Directors of CRASSH), our Faculty Advisors Tim Crane, Ruth Davis, Robin Kirkpatrick, James Montgomery, and Rowan Williams, as well as our external referees Duncan Large and Michael Schmidt. The excellent logistical and technical support we received from Esther Lamb, Glenn Jobson, Imke van Heerden, and Una Yeung was essential in ensuring the smooth running of the CCiT events. We would also like to thank the many speakers who participated in the discussions and workshops we organised: Adriana X. Jacobs, Theodor Dunkelgrün, Simone Kotva, Tony Street, Nicholas King, David Charlston, Danielle Sands, Duncan Large, Georgina Collins, Hephzibah Israel, Paul Russell, Angel Gurría-­Quintana, Orri Tomasson, Rosa Van Hensbergen, Manuela Perteghella, Hannah Conway, María Mencía, Peter Robinson, Olivia McCannon, Viviane Carvalho da Annunciação, Jennifer Harris, Jeremy Munday, Delia Chiaro, Paul Howard, Caroline Summers, Pauline Henry-­ Tierney, Jen Calleja, Olga Castro, Daniel Hahn, Gillian Lathey, Maria Nikolajeva, Anthea Bell, Lucile Desblache, Andrew Jones, Judi Palmer, Rachel Godsill, Helen Julia Minors, Lucy Taylor, Graeme Ritchie, Carol O’Sullivan, Catherine Boyle, Cristina Marinetti, Carole-­Anne Upton, Alfredo Modenessi, Adrià de Gispert, Andrew Rothwell, Francesca Billiani, Federico Federici, Rory Finnin, Shady Hekmat Nasser, Youssef Taha, Manuel Portela, Arnaud Regnauld, and Gabriel Tremblay-Gaudette. Many of the conversations that took place during and after these events refined our thinking about numerous issues that are considered at greater length in this book.

Acknowledgements  xiii

The ensuing chapters were first presented at a conference organised by CCiT that took place on 5–6 July 2018. We gratefully acknowledge the funding for this event that we received from CRASSH, the Institute of Modern Languages Research (IMLR), and the Cross-­Language Dynamics: Reshaping Community (CLDRC) Open World Research Initiative Project (the Translingual Strand).We also benefited greatly from the administrative expertise of Oliver Wright at CRASSH. After the conference, the authors were able to revise their chapters for this volume in the light of the resulting discussions. We thank them for the care with which they prepared those modified versions. We would also like to express our gratitude to Routledge for supporting this venture with such conviction from the outset. Louisa Semlyen, and our editorial assistants Hannah Rowe and Eleni Steck, provided us with effective guidance throughout the process; and the comments our proposed monograph received from several anonymous reviewers were invaluable. We would also like to thank the artist Gurpran Rau for kindly granting us permission to use one of her works as the image on the cover of this book. The editors and publishers would like to thank the following copyright holders for permission to reproduce the following material in this book: Raoul Dufy, 1877–1953 and Guillaume Apollinaire, 1880–1918, “Le Dromadaire” in Le Bestiaire ou cortège d’Orphée (Paris: Deplanche, Éditeur d’Art, 1911); woodcut on Holland Zonen wove paper; image: 205 x 195 mm (8 1/16 x 7 11/16 in.); sheet: 325 x 250 mm (12 13/16 x 9 13/16 in.); The Fine Arts Museums of San Francisco, gift of the Reva and David Logan Foundation, 1998.40.38.10; image courtesy of the Fine Arts Museums of San Francisco. “Il tramonto” in L’Allegria by Giuseppe Ungaretti (Milan: Mondadori, 1942). © 2015 Mondadori Libri Spa. While we have done our very best to identify and correct any lingering slips or blunders, we have surely missed one or two. If so, then nostra culpa.

Gunther Kress passed away, unexpectedly, while this book was in the final stages of preparation. We would like to express our gratitude for the encouragement and support he gave to this project from the very beginning, and we hope that the chapter he contributed is a fitting memorial. Monica Boria Ángeles Carreres María Noriega-­Sánchez Marcus Tomalin

INTRODUCTION Monica Boria and Marcus Tomalin

1. Translation in a multimodal world The current Wikipedia entry for translation begins with a confidently succinct definition: “[t]ranslation is the communication of the meaning of a source-­language text by means of an equivalent target-­language text.”1 That all sounds simple enough, but the simplicity conceals many lurking complexities. What does “meaning” mean in this context? What is a “text” exactly? And when is one text “equivalent” to another? Indeed, the difficulties and uncertainties concerning translation as a practice tend to proliferate the more one thinks about it, and especially when the source text contains nuanced intricacies of form and meaning. Here is the first stanza of François Villon’s famous “Ballade des dames du temps jadis,” which was one of the poems in his collection Le Testament (1461): Dictes moy ou, n’en quel paÿs, Est Flora, la belle Romaine, Archipiadés, ne Thaÿs, Qui fut sa cousine germaine, Echo parlant quand bruyt on maine Dessus riviere ou sur estan, Qui beaulté ot trop plus qu’umaine. Mais ou sont les neiges d’anten? (Villon 1994 [1461], 74) If these lines of medieval French function as a source-­language text, then their meaning can be translated into a target-­language text, such as this English version by the Victorian poet and artist Dante Gabriel Rossetti: Tell me now in what hidden way is Lady Flora the lovely Roman?

2  Monica Boria and Marcus Tomalin

Where’s Hipparchia, and where is Thais, Neither of them the fairer woman? Where is Echo, beheld of no man, Only heard on river and mere – She whose beauty was more than human? – But where are the snows of yester-­year? (Rossetti 1870, 177) It is immediately apparent that “meaning” (at least in the rudimentary sense of semantic content) is not the only factor being taken into consideration here. Rossetti’s translation reveals an attentiveness to the rhyme scheme of his source material, for example. Following the conventions of Villon’s forme fixe ballade, Rossetti ensures that “way is” rhymes with “Thais,” “Roman” just about rhymes with “woman,” “no man” with “human,” while “mere” rhymes with the final syllable of the compound “yester-­year.” Should these structural characteristics be considered part of the “meaning” of the original, or can we separate the form from the content without compunction? If semantic equivalence between the source and target texts is the primary goal of the translation, then attempts to preserve the rhyme scheme are likely to thwart that purpose (e.g., a “way” is hardly a “pays,” and “the fairer woman” has very little to do with “sa cousine germaine”). So maybe a prose translation would have been more accurate (at least in some sense[s])? And what about the broader cultural context of the stanza? Is it an innocent act to produce an English translation of these lines that reads as if it originated in an English-­speaking culture rather than a French-­speaking one? Or should that kind of approach be viewed negatively, as perpetrating cultural hegemony, assimilation, and, possibly, subordination? Perhaps the translation should seek instead to convey the foreignness of its source material more overtly? After all, Gallic allusions to figures from Classical history and myth have distinctive cultural resonances that are not retained when expressed in English. And then there is the bewilderingly vast panoply of specific, yet intricately interconnected, socio-­cultural perspectives. Take gender-­related considerations, for example. Villon’s stanza about celebrated female figures refers particularly to Flora (a Roman courtesan) and Thaÿs (another courtesan, but a Greek one this time), as well as “Archipiadéa,” who is probably Alcibiades, the male general from fifth-­century BC Athens who was sometimes mistaken for a woman in medieval texts because his beauty had been praised by Boethius in his De consolatione philosophiae (Boethius 2001 [524], 66). All of these references are subsequently translated by Rossetti, a male translator working in the late nineteenth century when traditional notions of gender as a socio-­political category were starting to be re-­evaluated (e.g., the National Society for Woman’s Suffrage was formed in 1867). So, how should a translator respond, if at all, to those aspects of the source text that are inherently gendered? Rossetti, it should be noted, has replaced Villon’s “Archipiadéa” with “Hipparchia.” Since the latter was undoubtedly a woman, the wife of the Crates of Thebes, she does not complicate the catalogue by blurring

Introduction  3

binary gender boundaries and creating ambiguities. Should Rossetti’s handling of this be viewed positively, as the correcting of a regretful error in the source text, or negatively as a disguising of an intriguing complication? And the various issues summarised briefly here represent just a tiny sample of the many possible evaluative and interpretative options that confront any translator.2 Indeed, in 1987 Eliot Weinberger and Octavio Paz showcased the remarkable range and variety of translation by taking a four-­line poem by the eighth-­century Chinese poet Wang Wei and offering nineteen different versions of the text, most in English (ranging from literal to freer renderings), but some in French, Spanish, German, and modern Mandarin (Weinberger and Paz 2016 [1987]). The fact that so many theoretical and practical considerations intersect, intertwine, and (not infrequently) conflict virtually guarantees that no single target text will content everyone, which is precisely why Peter Robinson has described translation as “the art of the impossible” (2010, title page). And a fascination with the impossibility of the task helps explain why topics such as those summarised earlier, and countless more, have received extensive attention in translation studies over many decades now. Yet, in recent years there has been a growing unease about the boundaries that determine the scope of what has traditionally been classified as translation. For instance, when Vaslav Nijinsky choreographed L’Après-­midi d’un faune for Sergei Diaghilev’s Ballets Russes company in 1912, he was inspired both by Stéphane Mallarmé’s 1876 poem of the same name and by Claude Debussy’s symphonic work Prélude à l’après-­midi d’un faune that had premiered in 1894. This cluster of related artistic creations prompts uneasy questions. Can Debussy’s musical representation of Mallarmé’s text be called a “translation”? Does it communicate the “meaning” of its source material, even though a shift from writing to music has occurred? Is it acceptable to refer to the music as a “text” and, if so, does that include a performance/recording of the piece, or only the written score – or all of these? And how should Nijinsky’s ballet be classified, given that it involves a further transference of meaning from writing and music to dance? There is currently no consensus about which sorts of meaning (re)communications can legitimately be classified as instances of translation. And it is particularly unclear how these questions should be answered in situations where the forms of communication are unambiguously multimodal (e.g., they involve more than merely writing).3 These theoretical difficulties may have been introduced here via a discussion of a ballet from 1912, but the need for a focused consideration of translation in multimodal contexts is becoming increasingly urgent in the modern world as communication involving words, images, movement, gesture, music, and so on occurs with ever greater frequency. The first decades of the twenty-­first century have witnessed an unparalleled proliferation of interconnected social, economic, cultural, and technological changes that have already begun to transform the nature of human communication in discernible ways. Many of these developments are closely linked to the joint processes of globalisation and technological innovation. The increased flow of goods, people, ideas, and money across national boundaries has profoundly

4  Monica Boria and Marcus Tomalin

altered the composition of the prevailing social and semiotic environments. Corporations (for instance) have increasingly perceived the need to adapt their products, branding, and advertising/marketing strategies to the cultural specificities of local markets. Consequently, globalisation dynamics have intertwined with localisation practices, using both verbal and nonverbal resources, to target different regional audiences/consumers – and we encounter these strategies daily as we interact with websites, packaging, billboards, computer games, interactive adverts, and the like. In addition, the ubiquity of the internet, and the rapid burgeoning of social media, mobile phones, and other digital technologies, have engendered a cultural moment in which texts, images, and sounds regularly combine to convey complex messages. In their different ways, emojis, internet memes, the automatic captioning of live-­streamed online shows, and e-­literature are all illustrative examples of this broad trend. Nonetheless, as the discussion of Nijinsky’s ballet has already indicated, none of these things is especially new in kind, though the incidence of them is undoubtedly increasing. Multimodal objects are easily identified in any cultures from virtually any historical period, whether it is Chinese logograms in the seventh century BC, or twelfth-­century illuminated manuscripts, or illustrated children’s books, such as Lewis Carroll’s Alice’s Adventures in Wonderland (1865). Indeed, certain cultural objects from earlier periods consciously drew attention to the difficulties that arise when meaningful communications in one mode have to be re-­expressed in a different mode. In Laurence Sterne’s extraordinary novel The Life and Opinions of Tristram Shandy, Gentleman (1759– 67), the character Uncle Toby and his faithful manservant, Corporal Trim, have a long conversation in which they discuss bachelorhood and celibacy. At one point, having stated that “[w]hilst a man is free –,” Trim gives “a flourish with his stick thus –”:

FIGURE 0.1 

The squiggle flourish

Introduction  5

As the eponymous narrator remarks, “[a] thousand of my father’s most subtle syllogisms could not have said more for celibacy” (Sterne 1767, 17). The exchange here playfully alludes to the multimodal nature of the conversation. The physical flourishing of the stick in the air is a meaning-­bearing gesture, and the communicative difficulty arises when that meaning has to be represented in a non-­gestural mode (i.e., writing). Sterne is wittily highlighting some of the difficulties that occur when novelists seek to record realistic conversations in detail, since face-­to-­face human interactions have never only involved sequences of words. Corporal Trim’s gesture with his stick is a profoundly meaningful one in the context of the discourse, yet it needs to be seen in order to be interpreted; and the graphical squiggle that approximates the movement of the flourish is little more than a rough translation of the actual gesture. This is the kind of communicative exchange that Norwood considers in her chapter for this book (Chapter 8), when she reflects upon the process of writing “drawingly.” Multimodal literary examples such as this merely reflect the commonality of such interactions in real life, and the emerging field of multimodality studies has sought to provide analytical frameworks that facilitate the study of such phenomena. In particular, the influential work of Gunther Kress, Jeff Bezemer, Carey Jewitt, Theo van Leeuwen, and others has outlined a social semiotic approach for understanding how distinct modes such as speech, writing, gesture, image, and sound function as semiotic resources in order to facilitate the representation and communication of meanings. In recent years, multimodality has been studied extensively in relation to domains such as psychology, advertising, social media, storytelling, discourse analysis, literary criticism, and gaming (to name just a few; see Jewitt 2014). Curiously, though, despite its increasing prevalence, minimal attention has so far been paid specifically to the impact that multimodal communication is having, and/or is likely to have, upon the theory and practice of translation. One exception to this is audiovisual translation, which has been discussed fairly extensively since the late 1990s, with a particular focus on applications in the media and entertainment sector (e.g., film subtitling; see Pérez-­ González 2014a). Indeed, some translation theorists/scholars occasionally give the impression that multimodality only refers to audiovisual phenomena. Nonetheless, as the prior discussion has emphasised, audiovisual translation constitutes merely a single manifestation of a much broader cultural shift, one in which nonverbal modes play far more than simply a minor contextualising role. The neglect of other multimodal environments in relation to translation is unfortunate. This is partly because the greater prevalence of these forms of communication and meaning-­ making has already begun to destabilise certain time-­honoured translation-­theoretic paradigms, causing old definitions and assumptions to seem fusty and inadequate. For instance, the word translation is currently deployed to denote meaning transfer across different modes and also within the same mode. Some influential figures, such as Kress, have argued that these distinct transferences deserve a more fine-­grained analytical vocabulary. Transduction could be used to refer to the moving of meaning from one mode to another (e.g., a painting based on a poem), for example, while transformation could denote the transfer of meaning within the same mode (e.g., an English

6  Monica Boria and Marcus Tomalin

version of a French poem) (Kress 2010, 124–31). At present, though, there is no agreement about such terminological details, and the absence of a stable and broadly accepted critical vocabulary frequently undermines discussions involving theorists who approach multimodality from different disciplinary backgrounds. Jargon-­related anxieties of this kind do not generally arise in the traditional translation paradigms, which (as we have seen) usually concern the conversion of a source text (frequently a literary one) into a target text. A translation theorist might examine in detail, for example, how Rossetti Englished Villon’s poem – and, as suggested earlier, such an analysis might focus on domesticating/foreignising tendencies, and/or problems posed by humour and wordplay, and/or the relationship between the respective systems of metre and rhyme, the importance of the cultural context, and so on. Since the 1980s there has been a much greater focus on various cultural and sociological aspects of translation, and issues such as power, ideology, culture, race, and gender have been extensively considered in texts such as Lawrence Venuti’s The Translator’s Invisibility: A History of Translation (1995), Gideon Toury’s Descriptive Translation Studies and Beyond (1995), Luise von Flotow’s Translation and Gender: Translating in the ‘Era of Feminism’ (1997), and Susan Bassnett and André Lefevere’s Constructing Cultures: Essays on Literary Translation (1998), to quote but a few key works. More recently still, pioneering scholars such as Martin Heusser, Willard Bohn, Marta Arnaldi, Helen Julia Minors, and Lucile Desblache have begun to explore different kinds of text-­to-­music, text-­to-­dance, text-­to-­image, music-­to-­text, dance-­to-­text, and image-­to-­text as instances of meaning transference (Bohn 2001; Heusser et al. 2005; Minors 2013; Arnaldi 2016). Although some of these analyses draw upon well-­established traditions of ekphrasis and iconology that can be traced back through the centuries (e.g., to illuminated medieval manuscripts; or, more recently, to Gérard Genette’s notion of visual paratexts), others probe in new and provocative ways the very limits of those activities we may wish to classify as “translation.” Although this is still an inchoate interdisciplinary field, it has already started to inspire ground-­breaking analytical approaches that richly deserve more probing scrutiny. Mary Snell-­Hornby’s identification of four different genres of multimodal texts, for instance, still raises at least as many questions as it answers (Snell-­Hornby 2009, 44). Other, more recent studies have applied cognitive theories to describe how the process of translating illustrated texts differs from that involved in translating a “monomodal” text (e.g., Ketola 2016). However, the outcomes of such studies have proved inconclusive at best, primarily exposing the urgent need for further research. This book addresses such issues directly, making them the central focus of the discussion throughout. In order to provide some disciplinary context for the main chapters of this book, it is important to summarise some of the dominant trends both in translation studies and in multimodality studies, since, if these two domains are to interact in a more effective manner in the future, then some mutual comprehension and familiarity is necessary.We will begin with an overview of the development of translation studies, focusing in particular on recent trends, before offering a summary of the emergence of multimodality as an academic discipline.

Introduction  7

2. Trends in translation studies Since its emergence in Europe during the 1970s, the field of translation studies has grown exponentially and internationally, and its development as an academic discipline has been charted numerous times (e.g., Baker 2001; Munday 2010; Lambert 2013; Bassnett 2014). Therefore, only a brief outline of its history will be summarised here, to give readers unfamiliar with translation studies some understanding of how and why this field has developed, to supply key references for further reading about the different schools and approaches, and to intrigue those who wish to delve deeper. The main focus will fall on some of the central controversies and on the most influential research traditions that have prepared the ground for contemporary multimodal approaches. Theorising about translation has a long and distinguished history. In the West, it is usually traced back to Cicero and Horace (first century BC). St Jerome (fourth century) and Martin Luther (sixteenth century) are regarded as pivotal figures, and the German Romantics, especially Schleiermacher, later exerted a powerful influence (Munday 2010, 421–423). However, a formal academic discipline devoted to the study of translation only became institutionalised in the twentieth century. Its development was prompted by several factors, including momentous historical events (e.g., the reliance on translation at the Nürnberg trials), Cold War advances in computer technology (e.g., machine translation), and the emergence of linguistics as a promising new “scientific” field of academic enquiry. In an effort to map linguistic transfer, the focus was initially placed on written language and on notions of equivalence between source and target texts. Roman Jakobson’s early taxonomy of translation as interlingual, intralingual, and intersemiotic (1959) was highly influential (see Kress’s and Kaindl’s chapters in this book for further discussion), but other linguists, such as George Mounin in France and Eugene Nida in the United States, were also pioneers of the field. Mounin is widely regarded as the founder of traductologie, the discipline devoted to the study of translation. He was taught by the French structuralist linguist André Martinet, and his work Les belles infidèles (1955) is widely regarded as a founding manifesto of the discipline. Nida, in turn, drew upon Chomsky’s theory of generative grammar, and introduced a distinction between “formal” and “dynamic” equivalence in translation. The former refers to a rendering that prioritises the form and linguistic structure of the source text, whereas the latter attempts to produce a target text that achieves an equivalent effect on the intended audience (Nida 1964). As these examples indicate, the importance of linguistics as a shaping force of the nascent discipline cannot be underestimated. Indeed, until the late 1960s translation studies was generally seen “as a branch of applied linguistics” (Baker 2001, 279), even though it had become clear over the years that notions of linguistic equivalence were ill-suited to an activity that involved much more than language (in a narrow sense). Consequently, translation studies became established in the 1970s, and its academic scope opened up to other fields, such as literary theory and comparative literature. The so-­called Leuven group of international scholars argued that

8  Monica Boria and Marcus Tomalin

translation should be recognised as a field in its own right. Their excitement at the increasing relevance of translation worldwide was matched by their frustration at its marginality in academia. James Holmes introduced the term “translation studies” in 1972, and at a 1976 conference in Leuven a manifesto was drawn up to announce the remit of the new field which, in the words of another participant, “concerns itself with the problems raised by the production and description of translations” (Lefevere 1978, 234). The new discipline aimed to study translation – the activity and its products – in all its forms, to understand its underlying “norms,” and possibly establish some general principles, which would inform practical applications, for example in translator training, translation criticism, and the like (Holmes 1988 [1972]).4 It was seen as being an inclusive field with a natural tendency towards interdisciplinarity, which involved not only scholars from different subjects but also professional translators. The hope was to foster a research culture where theory and practice would speak to each other and to the outside world. The success of this new academic discipline in the following two decades was largely due to the shift from linguistic notions of “equivalence” and “transference,” or ideas of  “ faithfulness” to the source text, towards an analysis of the socio-­historical context in which translation takes place, the political and ethical issues that impinge on it, and the role (and status) of the translator (Berman and Porter 2014, 3). Several key scholars contributed to these developments, both from a theoretical perspective and by taking practical measures to establish the field within academia and beyond. Susan Bassnett’s volume Translation Studies, published in 1980 (and now in its fourth edition), provided a fascinating and accessible overview of the important role of translation in Western literary and cultural history (Bassnett 2002). In outlining the various theories that informed its practice, Bassnett brilliantly demonstrated the rightful claim of the discipline to its place as an autonomous field of study. She has since been a strong advocate of the need for theory and practice to be in constant dialogue, and for a shift away from narrowly language-­based approaches towards broader culture-­based translation strategies. As Bassnett and Lefevere put it, “translation, like all (re)writings is never innocent. There is always a context in which the translation takes place, always a history from which a text emerges and into which a text is transposed” (Bassnett and Lefevere 1990, 11). Important contributions to the so-­called “cultural turn” in translation studies in the 1980s came from the Manipulation Group, with Itamar Even-­Zohar, Gideon Toury, José Lambert, and Theo Hermans. These scholars came from very different linguistic and academic backgrounds, and their theories embedded the study of translation in its socio-­ cultural context, presenting it as a phenomenon that influences the literary system it enters. Hermans has summed up some of their central ideas concisely: a view of literature as a complex and dynamic system; a conviction that there should be a continuous interplay between theoretical models and practical case studies; an approach to literary translation which is descriptive, target-­ oriented, functional and systemic; an interest in the norms and constraints that

Introduction  9

govern the production and reception of translation, in the relation between translation and other types of text processing, and in the place and role of translations both within a given literature and in the interaction between literatures. (Hermans 1985, 10–11) Prompted by similar interests, Mary Snell-­Hornby theorised an integrated approach involving linguistics and literary studies, dismantling notions of equivalence as illusory and misplaced, and advocating the importance of the translator’s cultural competence alongside their linguistic skills (Snell-­Hornby 1988). As this suggests, translation studies was partly the outcome of those winds of change that blew through universities in Europe and North America from the late 1960s onwards, and this partly explains why it has so frequently sought to “challenge orthodoxies” (Bassnett 2014, 25). The restructuring of higher education provision that followed those years of protests undoubtedly offered opportunities for new subjects to emerge, especially within the humanities and the newly established social sciences. The process was, of course, different in different countries, and despite the international make-­up and vocation of translation studies, the extent of the influence exerted by post-­structuralism, feminism, anthropology, postcolonial approaches to literature, and semiotics, varied considerably. The fact that translation, along with modern languages, had traditionally occupied an ancillary position within the Anglo-­American context placed translation studies at the margins of academia. It had to fight against ingrained prejudice to assert itself, and this lent the discipline an anti-­establishment air that set it apart from its European counterparts (Munday 2010; Bassnett 2014, 26). Lawrence Venuti (1995, 17) was one of the earliest and most vocal scholars and translators to expose Anglo-­American ethnocentrism and its conservative attitudes towards translation. In discussing “domesticating” practices that favour readability, obscure the inherent alterity of texts, and undermine the role of the translator, he exposed the insular, imperialistic culture of the United States, where translations (to date) barely reach 5% of publications. In the UK the situation is not dissimilar, with translations totalling just over 3% of publications, though recent statistics suggest that percentage is starting to increase (Anderson 2019). Foreign literature in translation is often segregated from English literature, frequently confined to different shelves in bookstores, unlike other countries (such as France or Italy) where Balzac, Calvino, and Dickens can usually be found in the same sections. Unsurprisingly perhaps, in France one in nine books is a translation, and the ratio is even higher in Italy, especially in children’s literature, where translations represent over 40% of the total titles for young and young-­adult readers (ISTAT 2018; Armitstead 2019). European (and bilingual Canadian) culture has traditionally been much more open towards translation, within the publishing industry and in society at large, with a long tradition of institutional investment in translators’ training and research (Lambert 2013, 9). Due to its hybrid nature as both an applied and a speculative subject, as well as its interconnections with other disciplines and schools of thought, translation studies

10  Monica Boria and Marcus Tomalin

has been able to respond more effectively to the challenges and opportunities of globalisation than many other disciplines – and it remains a vital academic domain. The vast number of available journals, national and international symposia, associations, federations (e.g., CETRA, IATIS, EST, ATISA), research centres, masters and doctoral programmes, and so on, all testify to the success of this “interdiscipline” (Snell-­Hornby 2006) that has explored its subject matter from such a wide range of theoretical perspectives. From postcolonial contexts to gender and sexuality studies, translating and translations have been scrutinised extensively to reveal the power imbalances, the economic and political inequalities, the gender hierarchies, and/ or heteronormative biases that have influenced cultural and linguistic representations. As the study of translation has shifted from the search for an ideal linguistic equivalence to a pervasive focus on difference, literary writing and translation have increasingly been caught in a complex tangle of power asymmetries. From differences based on language (global/local; national/regional) to those pertaining to culture (dominant/marginal; Eastern/Western), including issues of gender politics, the role of writers and translators in the construction of meaning and identity has attracted much scholarly attention. To take just two prominent examples, seminal work in feminist translation was pioneered by the Canadian scholars and translators Luise von Flotow and Sherry Simon (von Flotow 1991, 1997; Simon 1996), while research into the strategies of postcolonial translation provided the focus for Tejaswini Niranjana, Harish Trivedi, and Susan Bassnett (Niranjana 1992; Trivedi and Bassnett 1999). Studies such as these have teased out the implications of hegemonic discourses on the creation and circulation of literature, and have broadened the scope of the debate to include wider linguistic contexts and other disciplinary angles. The expansion of media and cultural studies since the late 1990s afforded translation studies unprecedented opportunities, and, as mentioned earlier, audiovisual translation (AVT) has emerged as the primary focus for multimodal studies of translation (Díaz Cintas 2009, 3). The field has increasingly attracted researchers from a large spectrum of academic backgrounds and countries (Dror Abend-­ David 2014). This poses the challenge of engaging with subject-­specific jargon and research methods, not to mention different academic traditions, and such complexities are usually a sign of genuine interdisciplinarity. Audiovisual translation’s increased prominence as a conceptual and aesthetic framework is testified by recent innovative studies such as Carol O’Sullivan and Jean-­François Cornu’s The Translation of Films, 1900–1950 (2019), which argues that translation has made a crucial contribution not only to the circulation of films but also to the art of cinema more generally. O’Sullivan and Cornu make their case by exploring the complexities of editing for international distribution, the aesthetic considerations of intertitles and subtitles, issues of censorship, and many other factors. In addition to these research areas, recent developments in digital technologies and the emergence of user participation, crowdsourcing, fansubbing, and other forms of amateur translation are providing audiovisual translation, and translation studies as a whole, with new sets of ethical and methodological questions (see the chapters

Introduction  11

by Luis Pérez-­González, and Elisabetta Adami and Sara Ramos Pinto in this book). Undoubtedly, the choice of English as the discipline’s lingua franca has been one of the contributing factors to its international reach and success. However, from the 1990s onwards, this field has widened its horizons beyond European and American perspectives. An increasing number of edited collections, anthologies, and readers have started to explore more diverse traditions of translation theory and practice, enriching the discipline with new visions and idioms, notably from East Asia, and perhaps exposing its own hegemonic bias (Bermann and Porter 2014, 6). Some scholars have even mooted a “post-­translation studies” agenda, to move beyond disciplinary boundaries towards wider transdisciplinary discourses on the translational nature of societies and individual identities (Arduini and Nergaard 2011, 8). Despite these developments, translation studies’ original disciplinary fields – linguistic and literary studies – have continued to influence translation-­ ­ based research, and, in different ways, the chapters in this volume by Matthew Reynolds and Marcus Tomalin form part of that time-­honoured tradition.Yet it is undeniable that in our increasingly intercultural, multilingual, and transnational world, translation as a discipline has truly come of age. The so-­called “translation turn” of the late 1990s saw other associated disciplines focus their attention on the theory and practice of translation, exploring its methods, and acknowledging its importance not only as a subject in its own right but also as the new paradigm that shapes our contemporary world. And that world is undoubtedly a multimodal one.

3. The rise of multimodality In some respects, the development of multimodality studies as a free-­standing discipline is a similar story. Over the past twenty years or so, it emerged from a cluster of interrelated interdisciplinary research domains which recognised that representation and communication involve far more than merely linguistic exchanges. As mentioned earlier, interest in this topic has arisen partly in response to the pervasiveness of globalisation and the burgeoning of digital media and technological innovation (e.g., Facebook, Twitter). It has already become a platitude to state that, in the modern world, communication frequently involves more than one mode. Internet memes combine writing and images, while vlog posts frequently involve speech, text, moving images, and music, all of which are coordinated to constitute and convey the meaning of the whole. In response to this, the term multimodality has been used to cover a wide range of approaches associated with the arts, the humanities, the social sciences, as well as engineering and artificial intelligence, and several different theoretical frameworks have been proposed. For instance, the centrality of multimodal communication was explored extensively during the 1990s by those seeking to create more effective human-­computer interfaces, while in a parallel tradition that drew upon Michael Halliday’s Systemic-­Functional Grammar, theorists like Anthony Baldry and Paul J. Thibault were outlining multimodal approaches to the analysis of texts (Bunt, Beun, and Borghuis 1998; Baldry and Thibault 2006).

12  Monica Boria and Marcus Tomalin

Having considered the various research traditions, Carey Jewitt, Jeff Bezemer, and Kay O’Halloran recently identified three core premises that characterise multimodal research in general (Jewitt, Bezemer, and O’Halloran 2016, 3): 1 Meaning is made with different semiotic resources, each offering distinct potentialities and limitations. 2 Meaning making involves the production of multimodal wholes. 3 If we want to study meaning, we need to attend to all semiotic resources being used to make a complete whole. Unsurprisingly, these premises come with several caveats, some of which are due to different terminological traditions and conventions. Not all multimodality theorists focus on “meaning making” overtly, for instance. Some claim that they are primarily interested in “multimodal discourse” or “multimodal communication,” while others deploy phrases such as “semiotic resource” rather than the term “mode.” While these variations reveal contrasting methodological and ideological allegiances, they can sometimes unintentionally distract from the underlying shared interest in forms of communication that involve more than merely language. Nonetheless, the field has acquired sufficient coherence that specialist journals have started to appear (e.g., Multimodal Communication was established in 2011) as well as devoted research series (e.g., the Routledge Series in Multimodality Studies), and these provide prominent fora for cutting-­edge research into multimodality. It is worth considering one particularly influential theory as a specific case study here – namely, Multimodal Social Semiotics (MSS). Although the origins of MSS can be traced back to the 1980s, it has already exerted a significant influence upon numerous academic fields. In essence, it is a branch of semiotics that seeks to understand how people communicate in specific social settings. Like several other theories of multimodality, it has deep roots in the work of Michael Halliday, and especially his monographs Language as Social Semiotic (1978) and An Introduction to Functional Grammar (1985). Halliday emphasised repeatedly that language should be interpreted “within a sociocultural context, in which the culture itself is interpreted in semiotic terms” (Halliday 1978, 2). Therefore, semiosis is not something that occurs merely in minds; it arises from social practices in a community. To facilitate his analysis, Halliday identified three related variables: field, tenor, and mode. Field refers to the subject matter or content of what is being discussed, thereby directing the focus towards what is happening, when it is happening, and to whom it is happening. By contrast, tenor denotes the social relation that exists between the participants in the interaction (e.g., father/daughter) and their purposes. It can therefore influence the communicative choices made by the interlocutors (e.g., the selection of a specific stylistic register). Finally, the term mode is used to refer to the channel of communication being used (e.g., speech, writing). These three variables combine to characterise any kind of situation in which meaning is communicated. Obviously, if more than one mode is being used simultaneously, then the interaction is (by definition) multimodal.

Introduction  13

The Hallidayan framework that underlies MSS was sketched out in Robert Hodge and Gunther Kress’s Social Semiotics (1988), and it has been gradually developed and refined in many subsequent publications, such as Kress and Theo van Leeuwen’s Reading Images (1996) and Multimodal Discourse (2001), van Leeuwen’s Introducing Social Semiotics (2005), Jewitt’s The Routledge Handbook of Multimodal Analysis (2009; 2nd ed. 2013), Kress’s Multimodality (2010), Bezemer and Kress’s Multimodality, Learning, and Communication (2015), and Jewitt, Bezemer, and O’Halloran’s Introducing Multimodality (2016), to name just a few. The fundamental analytical framework that MSS advocates necessarily draws upon aspects of older semiotic theories, but it reconfigures them anew. For instance, its insistent emphasis on the making and sharing of meaning enables it to be distinguished in various ways from most varieties of semiotics that trace their origins back to the work of Charles Sanders Pierce and/or Ferdinand de Saussure. MSS does not make use of Pierce’s tripartite classification of signs into the sub-­g roups iconic, indexical, and symbolic, for example, and unlike Saussurian semiology, it foregrounds the sign-­maker’s interest in the making of signs and meaning (using modes as semiotic resources), which leads to the striking conclusion that the relationship between form and meaning is never entirely arbitrary (Kress 2010, Chapter 4). As Kress and others have emphasised insistently, one of the underlying convictions that has guided MSS research from the very beginning is that multimodal interaction has always been the “normal state of human communication” – but, arguably, that claim is truer of the modern world than it has ever been (Kress and van Leeuwen 2001, 21). The different modes can be viewed as “semiotic resources which allow the simultaneous realisation of discourses and types of (inter)action,” and they have different affordances – that is, particular potentialities and constraints that impact the making of signs in representations (Kress and van Leeuwen 2001, 21). This is perhaps best explained by means of an example. A still image, such as a nineteenth-­century oil painting, uses the logic of space (rather than that of time), and its affordances include things like the materials of the surface the image is created on (e.g., a canvas), the kind of paint deployed (e.g., oils), the manner in which the paint is applied (e.g., impasto), and so on. These specific affordances are not shared by other modes such as music or speech. Each set of modal affordances is determined by the ways in which meaning is created with particular semiotic resources, and the latter are inevitably characterised by their material, cultural, social, and historical development. In many respects, there is nothing especially new in all of this. It had long been recognised that (say) different art-­forms functioned in different ways, and the problematical word ekphrasis (from the Greek roots for “out” and “to speak”) is still frequently used to refer to situations in which a physical object is brought vividly to life for a reader/listener by being described in extensive detail using either speech or writing. However, MSS offers a robust investigative framework that facilitates the analysis of such phenomena. In particular, it offers a way of conceptualising ensembles of modes – that is, representations that use more than one mode in a purposeful manner, to create collective and interrelated meanings.

14  Monica Boria and Marcus Tomalin

Therefore, ensembles are said to be “orchestrated” by at least one meaning-­maker (Kress 2010, 157–58), and an attentive consideration of the orchestration processes emphasises the extent to which monomodal communication is the exception rather than the norm. Recognising this fact profoundly destabilises many of our most familiar logocentric presuppositions about communication (generally) and about translation (specifically). Inevitably, one consequence of a social semiotic theory of multimodality is that language is necessarily displaced from a position of centrality in the analytical framework – just as it has been in many different sub-­branches of translation studies. More specifically, it loses its privileged status as the primary agent of meaning-­making, and it becomes merely one of many possible ways of creating and communicating meanings. Indeed, for Kress, language does not even merit being classified as a “mode” in its own right, since it is an abstraction that disguises the very real differences that distinguish speech and writing as semiotic resources. As he puts it, collapsing speech and writing with their entirely different materiality into one category, thereby joining and blurring over the distinct logics of time and space, of sequence and simultaneity, exposes the implausibility of a mode called “language.” It is difficult to see what principles of coherence might serve to unify all these features. So I take speech and writing to be distinct modes. (Kress 2010, 86) This observation raises several important issues. When it is recognised, for instance, that various modes (of which speech and writing are but two) can all be used as semiotic resources to make and convey meanings, then it is necessary to reflect upon the vocabulary conventionally deployed to refer to different ways of carrying meaning from one mode (or genre, or ideological complex, and so on) to another. Traditionally, of course, the term translation has been reserved for this purpose: a poem originally written in (say) French is translated into (say) English. In the Western tradition (at least), the conventional terminologies used to denote this process often imply some kind of moving or carrying across of the meaning. The etymology of translation reveals a compounding of two Latin roots: “trans” (i.e., across, over) + “latus” (i.e., borne, carried) – and the same is true of words such as traduire (French; from the Latin “trans” + “duco” [i.e., to lead]) and übersetzen (German: “über” [i.e., over] + “setzen” [i.e., to put]; so with the compound sense of “to pass or cross over”). Although this vocabulary generally refers to situations in which the meaning is moved from one mode to the same mode (i.e., from written French into written English, as in the Villon-­ Rossetti example discussed earlier), translators do, in practice, routinely deal with linguistic and non-­linguistic elements (e.g., lexical and sentential meaning, metre, rhyme scheme, page layout, font type). Be that as it may, the non-­ linguistic elements become even more apparent when the meaning-­material is

Introduction  15

moved across modal boundaries. For example, compare Giuseppe Ungaretti’s 1916 poem “Tramonto” (“Sunset”) with a digital image inspired by the text that was created in 2017:5 Tramonto Il carnato del cielo sveglia oasi al nomade d’amore

FIGURE 0.2 Tramonto

It is not possible (other than metaphorically) to analyse the image in terms of its lexis, syntax, morphology, and so on (though some images may contain some words and/or numbers), since the affordances of writing and image are so different. Nonetheless, we might still wish to classify this carrying over of meaning as constituting an instance of “translation,” even though it involves more than one mode. Certainly, depending on our “reading” of the image, some elements seem to be quite closely associated with the desert scene described by the text: a sunset, palm trees – perhaps even a nomad in a tent? As mentioned earlier, Kress has suggested that the high-­level term translation is too vague to be helpful in cases such as these and should be divided into distinct sub-­types. Therefore, the digital image inspired by “Tramonto” could be classified as a transduction of the poem-­text, since the meaning-­material has been moved from one mode to another – i.e., from writing to image (Kress 2010, 124–31). By contrast, Rossetti’s English version of Villon’s poem would constitute a transformation, since the meaning-­material has been moved from one mode into the same mode. In cases like the latter, a change of culture is involved (i.e., a French cultural context is replaced by an English cultural context), even though the mode remains the same, and this suggests that terms such as “intracultural” and “intercultural” are also pertinent (Davies 2012). There are, of course, many obvious differences between written French and English (e.g., syntax, vocabulary, alphabet, diacritics) but, in the MSS framework, the mode remains constant nonetheless. Although most multimodality theorists recognise the importance of being more precise about these distinctions, they do not always agree about the preferred terminology. Klaus Kaindl favours the terms “intermodal” and “intramodal” translation respectively, though the nature of the distinction is essentially the same (Kaindl 2013, 261–62). It is important to note, though, that the words used to sub-­classify translation always reveal something about the underlying metaphors and conceptual models that guide our thinking about the essential activity. As mentioned previously, terms that use the Latinate

16  Monica Boria and Marcus Tomalin

“trans” root all imply movement of one kind or another, and caution is required to ensure that this does not restrict the way we think about the underlying processes involved. Other languages offer different ways of understanding what we do when we translate. In Finnish, for example, the verb kääntää means “to translate,” but its root meaning is “to turn.” So, rather than being carried across a barrier, or from one place to another, translations in Finnish turn something over, to reveal another side (Chesterman 1997, 60–2). While there is still some kind of movement here, it does not involve transportation or relocation. This topic is discussed in greater detail by Kress in his chapter in this book (pp. 30–2). Distinguishing these different sub-­types of translation, and reflecting upon how meaning is communicated, compels us to look at familiar processes and classifications afresh. For example, if speech and writing are distinct modes (as Kress argues), rather than being merely distinct manifestations of the same mode (i.e., language), then the carrying, or turning over, of meaning from one mode to another should not be handled in a privileged manner. To revisit an example considered earlier, if Stéphane Mallarmé’s written poem L’après-­midi d’un faune is read aloud in French, then the meaning is moved, or turned, from the mode of writing to the mode of speech, and therefore the very act of speaking the text must presumably be classified as an instance of intermodal translation (i.e., transduction). This would place it in the same category as a performance of Debussy’s symphonic work Prélude à l’après-­midi d’un faune, since the latter is a musical re-­telling of Mallarmé’s written text. The fact that these two instances – the reading aloud of the poem and the performance of a composition based on the poem – fall into the same basic category should cause us to pause and reflect, since this classification destabilises most traditional analyses of translation. Despite the conspicuous differences between speech and writing that Kress accurately identifies, it remains true that there are more direct mappings between certain affordances associated with these two modes (e.g., the written and spoken forms of the same noun; the written and spoken forms of the same sentence) than exist between the affordances of writing and music (e.g., is there a direct correspondence of any kind between a specific noun or sentence in the written text and particular chords, phrases, modulations, cadences in the music?). Seemingly, some modes facilitate the intermodal communication of meaning more effectively than others, which in turn implies that it should be possible to elaborate an ontology of the various modes that specifies the nature of the relationships between them, as Tomalin suggests in his chapter for this book (pp. 138–42). Despite all of this, translation theorists have only really begun to engage with multimodality (whether MSS or other varieties) as an academic discipline.While, as mentioned previously, there is a fairly extensive literature on the topic of audiovisual translation, there have been very few attempts to probe the broader implications of multimodality for translation studies as a heterogeneous whole. Klaus Kaindl is one prominent figure who has recognised the need for such an engagement. He has written at length about the multimodal aspects of translating comics (a topic subsequently discussed by Federico Zanettin, Michał Borodo, and others), and he also contributed an important chapter on multimodality and translation to the

Introduction  17

Routledge Handbook of Translation Studies (2013), which sketches some of the intellectual background while also identifying significant research areas (Kaindl 2004, 2013; Zanettin 2008; Borodo 2015). Tellingly, he concludes that chapter by remarking that “translation studies has to develop appropriate investigation instruments for non-­language modes” (Kaindl 2013, 266). Another chapter-­length summary of multimodal translation, this time by Elisabetta Adami, appeared in The Oxford Handbook of Language and Society (2016), while, more recently, Sara Dicerto (2018) has attempted to elaborate a framework for analysing translation from a multimodal perspective, which places particular emphasis on the often unacknowledged role that pragmatics plays in such communicative situations. The burgeoning interest in these topics is also becoming increasingly manifest in academic journals. In 2013, Carol O’Sullivan and Caterina Jeffcote edited a special edition of the Journal of Specialised Translation devoted to the topic of “Translating Multimodalities,” while a few years later Catalina Jiménez Hurtado, Tiina Tuominen, and Anne Ketola edited a volume of Linguistica Antverpiensia which explored “Methods for the Study of Multimodality in Translation” (Jiménez Hurtado, Tuominen, and Ketola 2018). This all suggests that Luis Pérez-­González was quite correct when he remarked a few years ago that “[m]ultimodality is bound to become even more central to translation scholarship in future years” (Pérez-­González 2014b, 127). Several subsequent studies have focused on the multimodal implications of translating advertisements, illustrated technical texts, and the front covers of books (Pan 2015; Ketola 2016; Li, Li, and Miao 2018). As these examples suggest, the convergence between translation studies ad multimodality studies is already underway, and this volume overtly seeks to encourage and facilitate more extensive interactions.

4. An overview of this book The previous sections have summarised some of the key developments in both translation studies and multimodality studies that have characterised those respective disciplines during the last twenty years or so. They have also indicated some of the ways in which these two important research fields have started to merge and mingle. This book seeks to foster and facilitate such connections by offering insights into cutting-­edge research trends and by identifying specific topics that merit more extensive consideration in the future. As one of the first book-­length studies of translation and multimodality, it provides an unprecedented opportunity to examine some of the distinctive practical and theoretical challenges that confront those who create and/or consume translations in multimodal environments. In Chapter 1, Gunther Kress offers an authoritative view of translation from the perspective of MSS. Starting with a game in which people produce written descriptions of images that subsequently prompt the (re)creation of the original images, he outlines a framework for thinking about translation as a “transposing” of meaning. The discussion seeks to offer a preliminary account of particular site-­specific mode-­resources for making meaning and of the site-­specific mode-­ resources for transposing meaning. Similar themes are addressed by Klaus Kaindl

18  Monica Boria and Marcus Tomalin

in his chapter, but from a slightly different analytical perspective – one in which multimodality is viewed as a form of academic “rock ‘n’ roll.” After sketching in some of the intellectual background that associates contemporary approaches to multimodality with semiotic theories of “multimediality” from the 1970s, the focus shifts to exploring the way in which modes are unavoidably intertwined with both medium and genre. The core insight is that mode, medium, and genre are three building blocks that form the basis for a translation-­theoretical approach that serves to overcome the language-­centeredness of translation studies, and which enables us to view translation as a modal, medial, and generic practice. As a specific case study, Kaindl considers the reworking of Elvis Presley’s song Hound Dog (1956) in the German Schlager popular-­music genre, and the detailed analysis shows how the German version is characterised by changes that were made at the generic, medial, and modal levels. Building on some of the topics raised by Kress and Kaindl, Elisabetta Adami and Sara Ramos Pinto present an ambitious joint semiotic and translation research programme that seeks to outline ways in which both multimodality studies and translation studies can share their respective domains of expertise to address a series of disciplinary gaps that can only be effectively filled through a transdisciplinary enterprise. In essence, they encourage us to develop a deeper understanding of meaning-­making practices in a changed, increasingly transnational, social semiotic landscape. The kind of research they envisage requires methodological developments and disciplinary integration to facilitate the exploration of such things as the circulation of multimodal products, the practices of multimodal text production and translation, and the practices of (re)interpretation and re-­ signification. In a similar spirit, Luis Pérez-­González’s chapter concentrates on the evolving role of subtitling in the modern media landscape, specifically charting how this form of translation was gradually freed from the constraints of the film industry and came to serve more democratic forms of media “prosumption” associated with digital culture (e.g., Chinese danmu). In particular, Pérez-­González traces the shift from an ontology of referentiality, based on the hegemonic narrational regime, to the ontology of deconstruction that characterises the digital media ecology. He goes on to argue that, where ordinary rhetors have gained greater visibility and agency, their performance of citizenship involves the deconstruction of representation by exposing the cultural and social make-­up of multimodal ensembles. Adopting a markedly different approach, Matthew Reynolds situates his chapter in the long tradition of literary translation by taking a specific work of literature as his starting point – namely, Dante’s Divina Commedia (1320). In particular, he focuses mainly on one short passage from Paradiso (canto 26, line 134) in which a vertical (or slightly slanting) line is presented as the original name for God. This small detail raises non-­trivial problems for translators. Does the symbol indicate the vowel “I” or the numeral “1,” or is it something else entirely – something non-­ linguistic perhaps? The use of this vertical line also problematises the (multi)modal character of the poem, since the image appears as part of the written poem, and

Introduction  19

therefore it is not clear how it should be pronounced. These translation-­related uncertainties prompt a probing consideration of the classificatory problems that multimodal frameworks present. For instance, if we are unable to delineate a solid boundary around what constitutes a mode such as “speech” or “writing,” then we will inevitably struggle to define what modes actually are – and that in turn potentially destabilises some of the analytical presuppositions frequently adopted in multimodal studies. Similar concerns are examined in Marcus Tomalin’s chapter, which concentrates on the translation difficulties posed by livres d’artistes – that is, publications which purposefully combine texts and images to create multimodal ensembles. The publication Le Bestiaire ou Cortège d’Orphée (1911) resulted from a collaboration involving the poet Guillaume Apollinaire and the French Fauvist artist Raoul Dufy, and Tomalin takes one “poem” from this text, “Le Dromadaire,” as a starting point. By analysing both the poem-­text and the associated woodcut image, Tomalin introduces a (quasi-­)mathematical formalism that specifies more precisely the way in which multimodal ensembles are transformed during various processes that can be referred to as instances of “translation.” The final two chapters in the book relate some of these topics to other kinds of multimodal practices. More specifically, Helen Julia Minors explores the roles and processes of translation in relation to twentieth-­and twenty-­first-­century music-­ dance works. The particular examples considered include Debussy’s Prélude à l’après-­midi d’un faune and the Nijinsky ballet the music inspired, the controversial Stravinsky-­Nijinsky collaboration Le sacre du printemps, and Erik Satie’s La Parade. The discussion of these works illustrates the significance of understanding the transfer of “meaning” between these arts for those actively involved in the creative process, as well as for those who subsequently interpret and analyse the multimodal ensembles produced. The discussion of these issues raises numerous questions regarding how meaning is produced within and between these temporal gestural arts, and it is argued that the interplay of musical movement and danced movement relies on various forms of translation, which require cognitive mapping, gestural interpretation, and an awareness of somatic experience. By contrast with this emphasis on music and dance, Tamarin Norwood concentrates on how a novel method of writing can be derived from a particular technique of “half-­blind” drawing. In her experimental chapter, she shows how this process of derivation can be understood as a form of multimodal translation in which a drawing method (i.e., the original “text” being translated) is extracted from its original mode and reconstituted in the mode of writing, thereby offering a method of writing “drawingly.” This brief summary hopefully gives some idea of the range and scope of the issues this book addresses. In all cases, the discussions presented here are not the last word on any of the topics considered – quite the opposite, in fact. It is hoped that the core ideas and approaches outlined in the following chapters prompt others to reflect upon the relationship between translation and multimodality more profoundly and extensively. If that occurs, then this book will have more than served its purpose.

20  Monica Boria and Marcus Tomalin

Notes 1 https://en.wikipedia.org/wiki/Translation (last accessed on 2 September 2018). 2 The theoretical issues briefly alluded to here are all addressed in classic works gathered together in collections such as Schulte and Biguenet 1992; Weissbort and Eysteinsson 2006; Venuti 2012. 3 Some of these issues are addressed further by Helen Julia Minors in her chapter for this book (pp. 166–170), where she discusses L’Après-­midi d’un faune specifically. 4 For a discussion of Holmes’s project and the way it influenced the development of the discipline, see Malmkjær (2013). 5 The poem can also be translated as follows: “The flesh-­coloured sky awakens oases in the nomad of love.” The image was created by Monica Boria and Marcus Tomalin at the Translation and Multimodality conference that took place at the Centre for Research in the Arts, Social Sciences, and Humanities, at the University of Cambridge, on 26 May 2017.

References Abend-­David, Dror, ed. 2014. Media and Translation: An Interdisciplinary Approach. New York and London: Bloomsbury. Adami, Elisabetta. 2016. “Multimodality.” In The Oxford Handbook of Language and Society, edited by Ofelia García, Nelson Flores and Massimiliano Spotti, 451–72. Oxford: Oxford University Press. Anderson, Porter. 2019. “Nielson Reports Translated Literature in the UK Grew 5.5 Percent in 2018.” Publishing Perspectives, March 6. Accessed April 20, 2019. https:// publishingperspectives.com/2019/03/nielsen-­reports-­t ranslated-­l iterature-­i n-­u k-­ grows-­5-­percent-­in-­2018-­booker/. Arduini, Stefano, and Siri Nergaard. 2011. “Translation: A New Paradigm.” Translation: A Transdisciplinary Journal, Inaugural Issue: 8–16. Armitstead, Claire. 2019. “Mirror Writing: Authors and Their Translators.” The Guardian Review 6: 7–11. Arnaldi, Marta. 2016. “Moving Wor(l)ds:Translation as Transformation in Poetry and Dance.” Logios Journal 1: 35–46. Baker, Mona. 2001. “Translation Studies.” In Routledge Encyclopedia of Translation Studies, 2nd ed., edited by Mona Baker and Kirsten Malmkjær, 277–80. London and New York: Routledge. Baldry, Anthony, and Paul J. Thibault. 2006. Multimodal Transcription and Text Analysis: A  Multimedia Toolkit and Coursebook. London: Equinox. Bassnett, Susan. 2002. Translation Studies, 3rd ed. London and New York: Routledge. ———. 2014. Translation. London and New York: Routledge. Bassnett, Susan, and André Lefevere. 1990. “Introduction: Proust’s Grandmother and the Thousand and One Nights: The ‘Cultural Turn’ in Translation Studies.” In Translation, History and Culture, edited by Susan Bassnett and André Lefevere, 1–13. London and New York: Cassell. ———. 1998. Constructing Cultures: Essays on Literary Translation. Clevedon: Multilingual Matters. Bermann, Sandra, and Catherine Porter. 2014. “Introduction.” In A Companion to Translation Studies, edited by Sandra Bermann and Catherine Porter, 1–27. Chichester: Wiley-­Blackwell. Bezemer, Jeff, and Gunther Kress. 2015. Multimodality, Learning, and Communication: A Social Semiotic Frame. London and New York: Routledge.

Introduction  21

Boethius. 2001 [524]. Consolation of Philosophy. Translated by Joel C. Relihan. Indianapolis and Cambridge: Hackett Publishing. Bohn, Willard. 2001. Modern Visual Poetry. Newark: University of Delaware Press. Borodo, Michał. 2015. “Multimodality,Translation, and Comics.” Perspectives: Studies in Translation Theory and Practice 23 (1): 22–41. Bunt, Harry, Robbert-­Jan Beun, and Tijn Borghuis, eds. 1998. Multimodal Human-­Computer Communication: Systems,Techniques, Experiments. Berlin and Heidelberg: Springer-­Verlag. Chesterman, Andrew. 1997. Memes of Translation: The Spread of Ideas in Translation Theory. Amsterdam and Philadelphia: John Benjamins. Davies, Eirlys E. 2012. “Translation and Intercultural Communication: Bridges and Barriers.” In The Handbook of Intercultural Discourse and Communication, edited by Christina Bratt Paulson, Scott F. Kiesling and Elizabeth S. Rangel, 367–88. Oxford: Wiley-­Blackwell. Díaz Cintas, Jorge. 2009. New Trends in Audiovisual Translation. Bristol: Multilingual Matters. Dicerto, Sara. 2018. Multimodal Pragmatics and Translation: A New Model for Source Text Analysis. London: Palgrave Macmillan. Halliday, Michael. 1978. Language as Social Semiotic: Social Interpretation of Language and Meaning. Baltimore: University Park Press. ———. 1985. An Introduction to Functional Grammar. London: Edward Arnold. Hermans,Theo. 1985. “Translation Studies and a New Paradigm.” In The Manipulation of Literature: Studies in Literary Translation, edited by Theo Hemans, 7–15. London and ­Sidney: Croom Helm. Heusser, Martin et al., eds. 2005. On Verbal/Visual Representation. Amsterdam and New York: Rodopi. Hodge, Robert, and Gunther Kress. 1988. Social Semiotics. Ithaca, NY: Cornell University Press. Holmes, James. 1988 [1972]. “The Name and Nature of Translation Studies.” Translated! Papers on Literary Translation and Translation Studies, 67–80. Amsterdam: Rodopi. ISTAT. 2018. Produzione e lettura di libri in Italia. Report, December. Accessed April 19, 2019. www.istat.it/it/files//2018/12/Report-­Editoria-­Lettura.pdf. Jakobson, Roman. 1971 [1959]. “On Linguistic Aspects of Translation.” In Selected Writings II: Word and Language, 260–66. The Hague and Paris: Mouton. Jewitt, Carey, ed. 2014. The Routledge Handbook of Multimodal Analysis, 2nd ed. London and New York: Routledge. Jewitt, Carey, Jeff Bezemer, and Kay O’Halloran. 2016. Introducing Multimodality. London and New York: Routledge. Jiménez Hurtado, Catalina, Tiina Tuominen, and Anne Ketola, eds. 2018. “Methods for the Study of Multimodality in Translation.” Linguistica Antverpiensia 17. Kaindl, Klaus. 2004. “Multimodality in the Translation of Humour in Comics.” In Perspectives on Multimodality, edited by Eija Ventola, Cassily Charles and Martin Kaltenbacher, 173–92. Amsterdam and Philadelphia: John Benjamins. ———. 2013. “Multimodality and Translation.” In The Routledge Handbook of Translation Studies, edited by Carmen Millán and Francesca Bartrina, 257–70. London and New York: Routledge. Ketola, Anne. 2016. “Towards a Multimodally-­oriented Theory of Translation: A Cognitive Framework for the Translation of Illustrated Technical Texts.” Translation Studies 9 (1): 67–81. Kress, Gunther. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. London and New York: Routledge.

22  Monica Boria and Marcus Tomalin

Kress, Gunther, and Theo van Leeuwen. 1996. Reading Images:The Grammar of Visual Design. London and New York: Routledge. ———. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Hodder Education. Lambert, José. 2013. “The Institutionalization of the Discipline.” In The Routledge Handbook of Translation Studies, edited by Carmen Millán and Francesca Bartrina, 7–27. London and New York: Routledge. Lefevere, André. 1978. “Translation Studies. The Goal of the Discipline.” In Literature and Translation: New Perspectives in Literary Studies with a Basic Bibliography of Books on Translation Studies, edited by James Holmes, José Lambert and Raymond van den Broeck, 234–35. Louvain: Acco. Li, Long, Xi Li, and Jun Miao. 2018. “A Translated Volume and its Many Covers – A Multimodal Analysis of the Influence of Ideology.” Social Semiotics 29 (2): 261–78. Malmkjær, Kirsten. 2013. “Where Are We? (From Holmes’s Map until Now).” In The Routledge Handbook of Translation Studies, edited by Carmen Millán and Francesca Bartrina, 31–44. London and New York: Routledge. Minors, Helen Julia, ed. 2013. Music, Text, Translation. London, Delhi, New York and Sydney: Bloomsbury. Munday, Jeremy. 2010. “Translation Studies.” In Handbook of Translation Studies, edited by Yves Gambier and Luc van Doorslaer, 419–28. Amsterdam: John Benjamins. Nida, Eugene. 1964. Toward a Science of Translating:With Special Reference to Principles and Procedures Involved in Bible Translating. Leiden: E. J. Brill. Niranjana, Tejaswini. 1992. Siting Translation: History, Post-­structuralism, and the Colonial Context. Berkeley: University of California Press. O’Sullivan, Carol, and Jean-­François Cornu, eds. 2019. The Translation of Films, 1900–1950. Proceedings of the British Academy. Oxford: Oxford University Press. O’Sullivan, Carol, and Caterina Jeffcote, eds. 2013. “Special Issue on Translating Multimodalities.” Journal of Specialized Translation 20. Pan, Li. 2015. “Multimodality and Contextualisation in Advertisement Translation: A Case Study of Billboards in Hong Kong.” Journal of Specialized Translation 23: 205–22. Pérez-­González, Luis. 2014a. Audiovisual Translation:Theories, Methods, and Issues. London and New York: Routledge. ———. 2014b. “Multimodality in Translation and Interpreting Studies.” In A Companion to Translation Studies, edited by Sandra Bermann and Catherine Porter, 119–31. Chichester: Wiley–Blackwell. Robinson, Peter. 2010. Poetry and Translation: The Art of the Impossible. Liverpool: Liverpool University Press. Rossetti, Dante Gabriel. 1870. Poems. London: Ellis. Schulte, Rainer, and John Biguenet, eds. 1992. Theories of Translation: An Anthology of Essays from Dryden to Derrida. Chicago: The University of Chicago Press. Simon, Sherry. 1996. Gender in Translation: Cultural Identity and the Politics of Transmission. London and New York: Routledge. Snell-­Hornby, Mary. 1988. Translation Studies: An Integrated Approach. Amsterdam and Philadelphia: John Benjamins. ———. 2006. The Turns of Translation Studies: New Paradigms or Shifting Viewpoints? Amsterdam and Philadelphia: John Benjamins. ———. 2009. “What’s in a Turn? Fits, Starts, and Writhings in Recent Translation Studies.” Translation Studies 2 (1): 41–51.

Introduction  23

Sterne, Laurence. 1767. The Life and Opinions of Tristram Shandy, Gentleman,Vol. 9.York and London: T. Becket and P. A. Dehondt. Toury, Gideon. 1995. Descriptive Translation Studies and Beyond. Amsterdam and Philadelphia: John Benjamins. Trivedi, Harish, and Susan Bassnett, eds. 1999. Post-­colonial Translation: Theory and Practice. London and New York: Routledge. van Leeuwen, Theo. 2005. Introducing Social Semiotics. London and New York: Routledge. Venuti, Lawrence. 1995. The Translator’s Invisibility: A History of Translation. London and New York: Routledge. ———, ed. 2012. The Translation Studies Reader, 3rd ed. London and New York: Routledge. Villon, François. 1994 [1461]. François Villon, Complete Poems, edited by Barbara N. Sargent-­ Baur. Toronto: University of Toronto Press. von Flotow, Luise. 1991. “Feminist Translation: Contexts, Practices and Theories.” TTR:Traduction,Terminologie, Rédaction 4 (2): 69–84. ———. 1997. Translation and Gender: Translating in the ‘Era of Feminism’. Ottawa: University of Ottawa Press. Weinberger, Eliot, and Octavio Paz. 2016 [1987]. Nineteen Ways of Looking at Wang Wei (with More Ways). New York: New Directions. Weissbort, Daniel, and Ástráður Eysteinsson, eds. 2006. Translation:Theory and Practice: A Historical Reader. Oxford: Oxford University Press. Zanettin, Federico, ed. 2008. Comics in Translation. Manchester: St Jerome.

1 TRANSPOSING MEANING Translation in a multimodal semiotic landscape Gunther Kress

1. Preamble: the question of translation in the contemporary multimodal semiotic landscape In everyday interactions, banal or profound, actors in the contemporary social and semiotic world draw on a wide range of material means for making meanings (materially) evident – the modes of the semiotic landscape. In this chapter the aim is to explore what issues arise in moving from the dominance of one means (“language”) to a recognition of the potential equality of many means of making meaning evident (“the modes of multimodality”) and to consider the effects of that change on present conceptions of translation. Unlike, say, the chapter by Matthew Reynolds in this volume, the present chapter is not written from within translation theory. Its focus is a sketch of a paradigm shift, in terms of the issue at its core, moving from the centrality of “language” to a focus on “meaning”; and in terms of disciplines, a move from a linguistic to a semiotic frame. In that, it is part of the much larger changes affecting the global political, social, economic, and cultural landscapes, in Europe as elsewhere, which are unmaking centuries of relative stability. That brings the problem of “naming”: the names which had lent some stability to that world no longer adequately describe the contemporary social and semiotic landscape. Clarity in “naming” is required and, with that, the development of a terminology coherent and apt for that change. Throughout the chapter, I introduce requisite (semiotic) terms, tying them in to essential elements of the discussion.

2. A slightly odd take on translation: “Chinese Whispers” with a twist At a semi-­formal meeting, about twenty people – in the main, craftspeople/artists of various kinds, a few academics and others sprinkled in – are gently cajoled by

Transposing meaning  25

FIGURE 1.1 

The translation game

the Master of Ceremonies (MC) of the evening to participate in a game of “Chinese Whispers.” The MC has copied a range of different kinds of images – abstract, figurative, contemporary, traditional, “western” or, indeed, from anywhere on the globe. Members of the group are invited to select one of the images. Everyone has received an A4 piece of paper and a pencil. The MC tells the group to “produce a written description of ‘your’ image, in maximally thirty minutes” (see Figure 1.1). When the time is up, the images that had been handed out are gathered by the MC, together with the written descriptions. She then distributes these accounts randomly, one for everyone there. Around the room are tables with boxes of coloured pencils and crayons, to produce the drawings that are to be made on the basis of the written description each person has received. Everyone now tries to produce the image that corresponds with the written description they have. After maybe forty minutes of drawing, the MC calls a halt. She invites everyone to find the original image which had been the basis of the description. There is considerable interest in comparing the written descriptions with the original and with what has just been drawn from what had been written. Although everyone in the room is familiar with the game of “Chinese Whispers,” there is, nevertheless, astonishment at just how badly our notions of “communication” fit with a reality marked by difference of all kinds and at all points:

26  Gunther Kress

difference of personal “interest” of the participants, leading to difference in focus, leading to difference in selecting aspects from a message; based on differences in the “deeper” perspectives from which each of us selects elements to newly constitute the message. The game, as it is traditionally played, makes use of a shared meaning resource: speech. The “twist,” here, lies in the fact that there is no shared meaning-­resource – no shared “code” – for moving from image to writing and back to image. The building blocks of images are not words; and while there are relations between elements of the image, they are neither the elements nor the relations of the grammar of writing: no Subject-­Verb-­Object relations nor, for instance, indications of time as tense. The meanings at issue in the game – whether written or visual – do express the meanings of a particular community, yet they are made evident by entirely different material/semiotic means: more or less adequately, in line with the distinct affordances of writing and image. It raises the question: even though meanings of one social group are dealt with, the absence of a shared code for writing and image makes the activity, in many ways, quite similar to that of translation. (One difference being the absence, here, as yet, of relatively well understood assumptions of various kinds, which exist in language-­to-­language translation.) The traditional version of Chinese Whispers relies on the fact that the meaning resource of speech is shared by all players: it is used by each in relation to their differing interests and perspectives in the moment of playing. In the “twisted game” there is no shared meaning resource for moving between speech and image, even though there may be a broadly settled, general sense of shared meanings in that community of players. The “twisted game” brings out a strongly unsettling sense that writing and image each enable meanings to be made which cannot, however, easily be “shifted” from the one mode to the other – nor, actually, made at all in the other mode. Each mode offers profoundly distinct potentials for documenting meaning. Immediately the problem of “naming” emerges: in the move from one mode, writing, to a different mode, image, translation is not exactly what is the issue here. A term such as transduction, indicating a shift of a specific kind, suggests itself. The traditional version of the game makes clear that notions of “en-­coding” and “de-­coding” barely work even when using the shared resource of speech; in the “twisted” version it makes no sense. At a somewhat abstract level, writing and image realise aspects of social functions; they do not, however, share anything resembling a commensurate “code.” What questions – beyond those we know from the traditional version – arise from playing the “twisted version”? Any painter, sculptor, or composer will be familiar with the request “Could you give me some idea what this is about?” At times the response might be an exasperated “Have a good close look; do listen; reflect for a while; it’s all there.” In schools, in most contemporary textbooks and in most online resources, whether in Biology or Mathematics, in Geography or Physics, the majority of the page or the screen is given over to image, rather than to writing. Much the same – or even more so – is the case in an ever-­increasing

Transposing meaning  27

number of social domains. How can we expect this to work when as a society we have no real sense of the resources needed to reconstitute, to transpose a meaning which appears as an image – a diagram of a magnetic field, let’s say – with the resources of writing, of words in syntactic relations? Writing is still the privileged mode in important social domains (in schools, for instance, in the case of most exams; in legal domains; etc.).Yet the game does highlight the shortcomings of an approach based on the existing commonsense: writing does particular kinds of semiotic work; it does not and cannot do all of it. Despite centuries-­long certainties that “language” is fully capable of expressing all meaning that needs to be expressed, whether as writing or as speech, each is a partial means of making meaning. The same is true of image, as it is true of every means of making meaning. It is that which makes the case for a social semiotic multimodal approach. In the everyday, most messages still occur in face-­to-­face interaction; yet more and more frequently, messages come via the small screens of the devices we carry with us everywhere. There, image and writing usually appear together; though ever more frequently image is dominant or appears by itself. The majority of our communicational life is beginning to resemble versions of Chinese Whispers, with multiple twists. The huge project looming to be addressed is to bring to light the characteristics and the regularities of the wide range of meaning resources we now use every day, in still very new ways. That is an urgent issue already in just about every area of our lives. In relation to the interests of the group assembled that evening, the game highlighted, yet again, how difficult – impossible even – it is to describe meanings that inhere in “making” and in “things made”: that is, meaning beyond the presently recognised, canonical means. There is a need to find ways other than the limiting, only partially adequate, routes of writing or speech for making such meanings evident: giving attention and recognition to meaning-­makers in all areas of social life, to making, with all kinds of materials and to all things made. In the game played at the meeting, the MC kept the instructions simple: “Document the meanings made by your image as a written account, an account sufficient to allow the meaning of the original in your written account to be reconstituted as an image, by another member of this group.” Nothing could seem simpler, right? Yet what consequences loom if we do not develop relatively reliable means for such a simple task?

3. A brief sketch of a social semiotic theory and its tools for documenting and transposing meaning Social semiotics (Hodge and Kress 1988; van Leeuwen 2005; Kress 2010; Bezemer and Kress 2016) offers a clear focus on meaning and meaning-­making. It suggests both a frame and tools to account for the constitution of meaning. In that frame, it offers an account of translation seen as the transposition of meaning in the multimodal semiotic landscape of the contemporary social world. The term transposition offers

28  Gunther Kress

a conceptual/theoretical frame capable of dealing with changes in the making of meaning in all areas. “Position” – as an encompassing metaphor – is the crucial issue in all instances of making meaning: whether position in social environments, of semiotic resources, of personal “interest,” or of other significant factors. In what is still called translation, “position” is the issue. A rudimentary sketch of social semiotic theory is needed to make that case. Here, a brief outline of that theory is followed by an account of sign-­making as the core of that theory. A further step suggests that translation/transposition – in the now expanded sense of the transposition of all and any meaning – is an instance of communication and is accounted for in that overarching theory. Social semiotic theory takes it as given that meaning arises in social actions and interactions, communication being perhaps the most common of these. As discussed in the Introduction, the theory draws on the work of the semiotician-­ linguist Michael Halliday, and specifically on two assumptions: one (stated by him in slightly varying forms), that “language is as it is because of the functions it serves in society” (Halliday 1978, 18–19); the other, that language, as a full communicational resource, has to satisfy three (social) functions. One deals with meanings about “states of affairs in the world,” the ideational function; another deals with the meanings of the social relations of the participants in interaction, the interpersonal function. A third deals with the meanings produced by relations of elements within the text, as well as the relation of a text with its environment, the textual function. The first assumption asserts that the semiotic resources of a society are the outcome of their shaping in social (inter-­)actions. The second asserts that in order for any semiotic resource to be a fully functioning means for human communication, it has to deal with meanings which arise in each of these three distinct yet entirely integrated social domains. By substituting the phrase “semiotic resource” for “language,” and by bringing human agency clearly into the frame, Halliday’s statement becomes an encompassing social semiotic theory: “semiotic resources are as they are because of the functions they are made to serve in society.” It provides a workable frame for a social semiotic multimodal theory of meaning and meaning-­making. It asserts, among other things, that every community has a range of resources for making meanings evident: speech, gesture, gaze, writing, and others; that is, the modes of social semiotic multimodal theory. Further, the characteristics of the modes (their elements and relations) are the product of social semiotic action – of social and semiotic work, in a specific community – with and on the material “stuff ” of each mode. Modes and their constituent semiotic elements mirror both the interests of the makers and the potentials and limitations of the materials chosen for making meaning: their affordances (Gibson 2014). Examples of “material stuff ” are: movement of parts of the body against the (relatively) still upper torso in the case of gesture; air, in the case of speech; movement of the body in space, in the case of dance. Meanings are made constantly by every member of a community; hence, all meanings are the meanings of that specific community.

Transposing meaning  29

As a consequence, modes – together with their elements and the relations of elements – are repositories, traces of the history of meanings shaped in one community. The distinctly different affordances of each mode are the outcomes of ceaseless social semiotic work with and against the potentials and the resistances of the materials for making meaning. This work produces the semiotic/cultural resources of a social group. In these, each mode is both general (it meets the requirements of the three functions) and specific, given the different affordances of each mode for making meanings materially evident. All meanings made materially evident in any of the modes of a society bear a traceable affinity with others made evident in the mode, as well as with the meanings made evident in each of the modes of the society. “Affinities of meaning” has two aspects: a social-­(semiotic) aspect and affinities due to the shared materiality of a mode – a semiotic-­(social) aspect. This brief summary encompasses kinds of issues recognised in present theories and practices of translation, and it points to continuities of these with issues arising in the expansion of the field suggested here. It underpins the case of a paradigm shift from language to meaning, as well as that of a disciplinary shift from linguistics to semiotics. Mode, defined as material shaped in the history of social and semiotic work, makes the term and the uses of “language” problematic. Given that both the material (air, in the case of speech; surfaces and traces on these in the case of writing) as well as the histories of social semiotic work in each case are distinctly different, it is difficult to see how speech and writing can reasonably be subsumed under the one label, language. As a consequence, in the social semiotic multimodal approach outlined here, the term language is replaced by the terms speech and writing, each naming a distinct mode. One of the benefits of the move from the use of the term of language to that of mode is that whatever the semiotic resources of a community are – for instance, whether they do or do not use an alphabetic or syllabic or character-­based or any other kind of “script” – all are included in the one set of semiotic principles. All communities have developed semiotic modes requisite to social need and material potentials. All have and use modes. Crucially, the theoretical notions of meaning-­ making and of the transposition of meaning apply to all. Abandoning the term “language” introduces a need for careful, apt naming. We cannot, now, use a term that names a semiotic resource (e.g., “English, the language”) to name (even if implicitly) a social-­cultural-­political entity: as for instance in “The mode of gesture works differently in English than it does in French” (where “English” names both a mythical place, Englishland, and a semiotic resource). “English,” the semiotic resource, is shaped by the social practices of very different social-­cultural/political entities: in Nigeria, in Hong Kong, in England, in Scotland, on the East Coast of the US, in Wales, etc. In short, the signifier “English” is now a problematic homonym. Given the present lack of adequate names, circumlocutions will have to do: for instance, “the mode of writing in Nigerian English culture,” “the mode of gesture in Hong Kong English culture.”

30  Gunther Kress

The recognition of a wide range of modes, and an awareness of the changed significance in communicational uses of modes beyond those of speech or writing, forces a rethinking of issues around meaning. On the one hand, each mode, despite its differing affordances, provides means for realising meanings in each of the three functions of social semiotic theory. That is a general and unifying aspect of the theory. On the other hand, given the constant social work in a specific community on materials chosen for semiotic use, each mode has distinct semiotic characteristics. The affordances of speech, for instance, are an effect jointly of its temporal characteristics (the logic of time and sequence) interacting with the material potentials of sound (such as pitch, energy, duration, pace), both shaped in social semiotic work in particular environments. The characteristics/affordances of (still) image are the result of its spatial logic interacting with the potentials of mark-­making on a surface, together with the histories of its semiotic work. The mode of gesture combines the logics of spatiality and of temporality (given the movement of hands, fingers, arms, against the stable upper body of the signing person). In the well-­understood academic practice of transcription – from speech to writing in one community – there is a pretty well taken for granted assumption (not expressed in this way) that the target mode (writing) provides the transcriptional resource – letters usually (or, in some instances, lexical items/words) – by means of which the elements of the source mode – speech sounds (or lexical entities of speech) – can be newly constituted. Transcription provides a good example for documenting the manifold problems inherent in this seemingly straightforward practice: the alphabet being a notoriously bad resource for the task of newly constituting meanings spoken as meanings written. Transcription does, however, supply two general principles: one, that the resource to be used in the transposition of meaning is a resource available in the target mode (letters in this case; or lexical items in standard written form); and two, that the relation of the elements of the reconstituting/transpositioning mode to the elements of the originating mode is that of aptness more or less, hence metaphoric. That is, the “meanings” (the sonic quality) of the sounds of speech (as the signified) are made materially evident in the resource of letters (as signifiers), treated as apt means of achieving this. In other words, the relation between the signs of the source mode (here speech) and the signs of the target mode (here writing) is never one of equivalence, but always one of aptness, hence of metaphor. The meanings produced in one community can be realised by any of its modes. At one level they are the “same” meanings, even though realised in the affordances of the different modes of a community. If there is a need for a meaning such as “intensity,” it can be expressed (“be given material realisation,” “made materially evident”) through the affordances of each of the different modes. In the mode of gesture, “intensity” might be realised by pace and/or by the extent of the sweep of the gesture. In the mode of speech, “intensity” might be realised through greater energy, as loudness, or by lengthening a sound. In writing, “intensity” might be realised by bolding, or as the thickness of a line in handwriting, or by capitalisation. In both the modes

Transposing meaning  31

of speech and of writing, lexis is available as a means of indicating intensity, via use of words such as very, huge, and terrible. Such issues are dealt with in present conceptions of translation.What differs here is that different realisations of “one meaning” occur within one and the same community, and at times have to be expressed – made materially evident – in another of that society’s modes and its affordances. In that “other mode,” social semiotic work will have shaped the different material of that mode in forms apt to the users of that mode. An example might be a “gestural” unit or “the shot” of a kineiconic mode (Burn 2014). Hence “one meaning” has a distinctly different “appearance,” a distinct material realisation in the one community. This “different look” may have given rise to the notion of “different semiotics” (and of intersemiotic translation) to deal with different materialities: semiotics of film, gesture, image, music, dance, etc. The previous centrality of language – with the assumption of its capacity as the means of making all meanings evident – may have had a large effect on this: when language was seen as central and all-­encompassing, other means of making meaning might need to be dealt with by a different (semiotic) theory. Across societies, specific social semiotic work will have shaped the modes of speech, image, gesture, and writing in the distinct ways required by the source and the target society. Given the different development of modes in specific communities, the assumption of “same mode, same meanings” cannot be sustained. The realisation that semiotic features are differently realised across societies and languages (in the traditional sense) is taken for granted in practices of translation. However, the previous example (and the theory underpinning it) combines that with the recognition of a multiplicity of modes in each of the communities, each with distinctly different affordances given the material differences of modes. The principles are – in the main – of the same kind as those of translation across different “languages.” What traditional approaches to translation had raised (e.g., Jakobson’s 1959 “intersemiotic translation”), but not made central, are the means for the recognition of modes and the conditions of the entirely usual transposition of meaning involving all modes: whether in the same or across different societies or, not unusually, both. Social semiotic theory assumes a homology of modes in the source society matched by a homology of modes – distinct and different – in the target society. The difficulty lies in the likely absence of such apt transpositional resources by modes in the target society. The term transposition, proposed in this chapter, draws attention to two core issues. One is “position” in a social sense and the other is “position” in a semiotic sense: that of the material semiotic resources involved. That is, signs are made in specific environments, social and semiotic; they bear the traces of the environments of their making. “Position” in the second sense refers to the materiality of the mode in which meaning is made materially evident, in the signifier. That “position” has epistemological and ontological effects (as pointed out later). Both the initial maker of the sign and the person engaging with that sign in interaction are positioned socially and semiotically, though most likely differently in one or both.

32  Gunther Kress

To restate this slightly: neither a sign nor a multimodal ensemble of signs “has” a meaning. In their initial making in a specific environment and in the materiality of specific modes, the meanings of the sign maker are made materially evident. In engaging with the sign as a prompt, it is a meaning resource for the social semiotic actor who engages with the prompt. “Meaning” is not transposed: rather, the potentials for making meaning occur in two positions. In each, signs are made: once in initial making and once – as a new and different sign – in its “interpretation.” What has been transposed is a potential for making meaning, from one position to another. In each position, meaning is made according to the characteristics of the environment and the affordances of the modes. Meaning is never transported in the manner of a book that is parceled up and sent to me by Amazon. In translation it is assumed that the source and the target community have modes with the same (or at least similar) semiotic entities. They might both have modes with lexical items, grammatical (morphological and syntactic) forms, and both have larger textual organisations, genres maybe. One-­to-­one correspondence cannot be assumed. The theory needs to work equally well for societies which use modes based on the alphabetic “rendering” of speech and those which are not; for societies which have communicational resources such as so-­called Australian Aboriginal paintings; or be capable of dealing with the relation between a character-­based script and speech. Nearer to home would be the relation between means of documenting mathematical meanings and their transposition into the affordances of alphabetically rendered speech. To recap at this point: Halliday’s amended dictum “semiotic resources are as they are because of the functions they are made to serve in society” allows us to expand the present frame of “translation.” It makes it possible to bring the field as traditionally conceived together with the now essential task of dealing with transposition of meaning of all kinds in any situation into one coherent field. Modes realise meanings in all three functions. That is a unifying principle for meanings and modes, whether across a community or beyond that across all social organisations – groups, societies, ethnicities.What modes exist in a community, how and to what extent they are (materially) developed, which are communicationally foregrounded, differs from one social group to another. The theory and its functional requirements are the same.

4. The partiality of the semiotic resource of modes As a consequence of the differing materiality and the social semiotic histories of shaping, and hence of the difference in the affordances of modes, in a social semiotic account all modes, speech and writing included, are seen as partial means for realising meanings. While all modes meet the requirements of the three functions, they do so in terms of their different affordances. For example, the social and semiotic feature of coherence will need to be realised in all modes of a social group. If the mode is writing (in UK English, say), the entities

Transposing meaning  33

available to realise coherence include pronouns, (other) lexical items, syntactic forms, punctuation, type of font, and so on. These entities are not available in the mode of image. There, different semiotic features are available to realise coherence: colour, for instance (as colour palette), or placement of elements in spatial relations. In speech (in UK English), intonational features are used as cohesive features to produce coherence. In arranging a group of people for a photograph, placement in the group or proximity can be used as semiotic features – signifiers – to produce signs of coherence. These are instances of signifieds of the textual function being realised materially, in each case through different signifiers of cohesion and coherence in different modes.While all modes realise meanings in each of the three functions, their different affordances mean that signifieds (meanings) are made materially evident differently. This is where the notion of the partiality of modes arises: each mode meets the functional requirements differently, as Tamarin Norwood’s discussion in her chapter for this volume demonstrates. In each case, the difference matters. Through complementarity of signs in ensembles of different modes, the meanings intended are more likely to be achieved. One corollary of partiality is that modes tend to occur in multimodal complexes: whether to provide means for complementarity, or for additional as well as for parallel or divergent meanings in a multimodal complex. Take (as an invented example) a written element in a multimodal complex: “the trees stood out against the darkening sky.” Here the inceptive aspect “darkening” of the mode of writing (in UK English) might be accompanied by elements of the mode of moving image showing both “developing darkness,” together with uses of colour, all complementing, “underlining” the morphemic meaning of the “inceptive” aspect. The potential of foregrounding a mode (or two or three) exists in any multimodal complex, as do different uses attributed to and met by each mode. These are a matter of socially motivated design in specific instances, always including intentions of both rhetor and designer of the complex. Those who engage with the multimodal complex cannot, of course, know whether the rhetor and designer (whether as one or as two persons) felt that they had fully achieved the meaning they wanted to convey. That is true of all multimodal complexes, whether of a (largely) musical composition, a painting, a film, etc. Was Kandinsky satisfied when he had completed the painting entitled “Kossaks”? Complementation does not guarantee that meaning has been fully represented. Comprehension is always partial; interpretation is always a hypothesis. In relation to the transposition of meaning, whether within a society or across societies, given the different affordances of modes in different social environments, there is always a question whether and which of the potentials for creating meanings of modes, of all or some in a modal complex need to be transposed; or whether indeed in the process of transposition quite different modal complexes will need to be designed and produced. Although, as noted earlier, meanings are never transposed – the potentials for creating meaning are.

34  Gunther Kress

5. Sign-­making: interest, criteriality, aptness Sign-­making is meaning-­making; it is the core of semiosis. Signs are the constitutive elements of modes and of modal complexes as texts. Sign-­makers use available material-­semiotic resources, signifiers, from within specific modes, chosen as apt in relation to the signifieds of the social group in which the signs are to be made. Every sign (and therefore all multimodal sign-­complexes) tells us something about the sign-­maker. How she or he saw the world in the moment of making the sign is revealed in the choice of the signifier which was regarded as apt. The following example makes the point. A three-­year-­old boy, sitting on his father’s lap, draws a series of seven circles (see Figure 1.2).When he has finished, he says: “this is a car.” The question arises as to how this is or could be “a car.” While drawing, the child had said: “here’s a wheel, here’s another wheel, that’s a funny wheel . . . this is a car.” The criterial feature of a wheel was that it was circular; hence circles were apt signifiers for representing wheels. The criterial feature of “car” seemed to be that it had many wheels; hence the arrangement of seven circles seemed an apt signifier for representing “a car.” To represent wheels by circles rests on analogy: wheels are like circles; “aptness” is a feature of analogy. The outcome of the analogy is a metaphor: “a wheel is (like) a circle.” Similarly with the representation of “a car”: “a car is something with many wheels.” The car-­sign, and its meanings, is the result of a sequence of two metaphors. For this sign-­maker, the signifiers (the form) “circle” and “many wheels” are apt (that is, their material form and their histories of use make them suitable) to be the carriers of the signifieds of “wheel” and of “car.” We might ask further: why and how could, for this three-­year-­old, a circle be the signifier for a wheel, and how can wheels be the criterial feature for “car.” The answer to the first question seems self-­evident: both are round. “Roundness” provides the criterial feature for the analogy. As far as the second question is concerned, the answer might be that if we imagined the eye-­level view of a three-­year-­ old, looking at the family car (in this case, a 1982 VW Golf, with its prominent,

FIGURE 1.2 

Drawing by a three-­year-­old: “this is a car”

Transposing meaning  35

chunkily visible wheels, especially at the three-­year-­old observer’s height), we might conclude that his position in the world, literally, physically, cognitively, and affectively, might well lead him to see cars – as it did on this occasion – in that way. His drawn sign represents his “position,” broadly speaking his “interest,” arising out of his (physical, affective, cultural, social) position in the world at that moment, vis-­ à-­vis the object to be represented. Generalising, we can say that interest shapes attention to a part of the world which is in focus and acts as the motivation for (principles of) selection of that which is to be represented (Kress 1997). The point is that the interest (in the sense indicated) of the sign-­maker/meaning-­ maker determines, in the moment of its (material) realisation, what is taken as criterial about an entity. That which is taken as criterial will be the signified and becomes central in the making of the sign. The child’s drawing suggests and realises a view of a part of the world which is, for him, historically, socially, and culturally shaped. What the meaning-­maker takes as criterial determines what (s)he will represent about that entity or phenomenon. The drawing is the result of the child’s semiotic work in his engagement with a part of the world: it makes his distinct interest evident in his selection of the material signifier. Three points are crucial in this example. First, the relation between meaning/ signified and form/signifier is not arbitrary. It is motivated. That is, the form suggests itself to the sign-­maker because it is, for her or him at this point, the apt means to be the carrier of the meaning. The signifier (in its material form), taken together with the history of its prior uses (in ways known to the sign-­maker: here, as a personal history of two years of making constantly changing circular drawings) satisfies the interest of the sign-­maker in finding an apt material means of expressing the signified. Form and meaning exist in a relation of motivation. This sign, like all signs, whoever the sign-­maker, is constituted on the basis of a motivated relation of form and meaning. That process is no different in the transpositions of translation: the choice of signifier (whether sound, word, phrase, or larger textual entity) in the target mode is made on the basis of the same principles. Interest, of course, is (among other things) often professionally shaped. It is one means by which power enters into the transposition of meaning. The second issue, closely linked to the first, is that the sign – at the moment of its constitution – is based on that which is criterial to the sign-­maker about that which is to be represented. All signs represent that which is criterial about what is to be realised, materially, to be represented. In that sense the sign is always both full and partial: fully representing that which is criterial in the moment of sign-­making and partial in relation to the many other features which – from other perspectives, on other occasions, and of course for other sign-­makers – constitute the semiotic entity being reconstituted in being transposed. In every sign the relation between form and meaning is motivated by the interest of the sign-­maker. The sign is always both a “full” representation of the sign-­maker’s interest and only ever a partial account of some entity or phenomenon when seen from a different position,

36  Gunther Kress

such as professional or other considerations and demands, which shape attention and selection. The third point follows from the previous two: every sign is newly made. It is a crucial assumption in social semiotics that all signs, with no exception, are newly made in the moment of their making, in the manner just described. Signs are never used; signs are newly made. The prompting sign is taken as the signifier, to be interpreted. The new sign having been made internally, the signifier is transformed, however minutely, bearing the traces of its recent as well as those of all other uses. That is the case for all signs, made by any maker of signs, anywhere, however banal or portentous the sign might seem to be. The notion of “new making” – of novelty, creativity, “interpretation” – is bound to be problematic for traditional notions of “accuracy,” “precision,” “veracity” in mainstream practices and assumptions about translation. That is a fact to be recognised and dealt with in a theory of translation as transposition. Among others, it is a matter of what audience “holds the aces.” The first of these three issues permits making hypotheses about the interest of the sign-­maker: the sign is the trace of the semiotic work of selecting an apt signifier for the signified. The choice of signifier provides an insight into the sign-­maker’s principles underlying the sign-­making. When the sign is made in response to some prompt, it gives insight into the sign-­maker’s semiotic work of selecting elements from the prompt and of the semiotic work of making a sign that aptly constitutes – for her or him, at this moment – an apt response. The second issue, the partiality of the sign, provides, in the response, an indication of what was considered to be criterial by the sign-­maker about the prompt and what was not taken as relevant or criterial. The third point, the always new making of the sign, moves away entirely from still existent mainstream notions of communication (the notion of the faithful copy, the accurate decoding), of translation as an instance of communication in which the aim is producing faithful copies of the original – accurate replicas, as near as possible.

6. Communication in the frame of social semiotic theory In the sketch of the social semiotic theory presented so far, sign-­making as action is present, even if only implicitly, whereas interaction is not. Sign-­making, however, is always the action of a sign-­maker embedded in the social and semiotic world. In the absence of the social environment in which she or he is acting, there would be no indication of what has prompted the sign-­maker to make the sign, nor any indication for whom the sign has been or will be a prompt. The prompt is crucial to the making of a new sign. It represents the social characteristics of the interaction with an addressee, which shapes the prompt. In communication, the prompt originates with one of the participants in the interaction. It is a social and a semiotic phenomenon: as a message produced by one of the participants for another and as a prompt to engage with. Interest provides the recipient’s motivation for engagement and shapes attention. It leads to a framing

Transposing meaning  37

of aspects of the prompt and a consequent selection of elements in that frame for interpretation according to principles brought by the interpreter. The engagement, as with the car-­sign, is with that which the interpreter regards as criterial. In that process the participant makes her or his sign “internally,” leading to a reorganisation of the interpreter’s “inner” resources. In this theoretical frame, communication cannot be regarded as a matter of decoding: it always results in the new making of a sign. Communication has happened when there has been interpretation resulting in the new making of a sign. Communication is a process of the transposition of semiotic material. An example from a science classroom in inner London – 13-­to 14-­year-­old women – may help make the point. The example comes from a series of Biology lessons: the topic was “plant cells” (for details, see Kress et al. 2001, Kress 2010). The class has had four lessons on cells. There has been much talk by the teacher; images have been shown and drawn; an experiment has been conducted. In the fourth of these lessons, the class is back in their homeroom. The teacher asks: “Can anyone tell me something about plant cells?” One young woman raises her hand, and on being nominated says, “a cell has a nucleus, Miss.” The teacher says “Good! Susie, can you come up to the front and draw what you have just told us?” Susie walks up to the whiteboard, and Figure 1.3 is (roughly) what she draws. She has responded to the teacher’s request for a reconstitution of meaning, a transposition: most immediately a transduction from the mode of speech to the mode of image. The crucial point here is about mode, modal affordances, and the “drawing on” of resources for making meaning across modes – the twinned processes of transduction in transposition. The modes of writing or speech each prompt specific questions about “the world”: here the relation of cell and nucleus: different to those posed by image. The epistemological affordances of writing (and of speech, similarly) suggest questions such as: “How is the relation between the cell and the nucleus characterised?” “Is it as a possessive (‘a cell has a nucleus’) or as a spatial relation? (‘The nucleus is in the cell’)?” The epistemological and ontological affordances of image differ from those of speech or writing. They might prompt a question such as “Where in the cell is the nucleus

FIGURE 1.3 

Plant cell

38  Gunther Kress

located?” “What size, relative to the cell, is the nucleus?” Each mode brings specific epistemological and ontological commitments. It is not possible to draw a cell without placing the nucleus somewhere within the cell wall. A teacher might not pay too much attention to this absolutely unavoidable epistemological and ontological commitment, though a student in the science classroom is entitled to regard the teacher’s placement of the cell as criterial: an indication of certain knowledge. The distinct epistemological and ontological potentials of different modes are recognised across different communities. The carpenter’s sketch, the architect’s three-­ dimensional model, the police investigator’s drawn reconstruction, the researcher’s diagrams and transcripts, all “fix” (Kress 2000) a specific “take” on the matter in focus. Each exploits the distinct potential of modes to provide different insights into the world framed and in focus. This makes epistemological/­ontological questions an unavoidable issue in the constitution of meaning, and therefore in every act of transposition. If meaning comes into being as the result of the semiotic work of interpreting, we might say that “knowledge” comes into being only when it has been made materially evident. (It is necessary to distinguish my use of “interpreting” here from the specific sense of interpretation in the translation context, that is, as “impromptu oral paraphrase”). Knowledge in the spoken response “the cell has a nucleus” is ontologically different from the knowledge displayed in the drawing. That is a “take” beyond that of translation from one written text to another. It tends to be resisted by teachers, who say “I’m interested in knowledge; I’m here to teach Biology, not to worry about drawing or writing or whatever.” The cell example shows that as learners make signs, they are required to transpose potentials for meanings to be made in one (or more) modes or ensemble of modes, in one environment to other modes or ensembles of modes in a different environment: from laboratory to classroom, to examination room, with pen and paper as tools. It matters how a teacher labels this practice. Each shift entails transposing the potentials for meaning and reconstituting the meaning in terms of the affordances of the modes used and the site in which it is done. That is always shaped by the designer’s interest and agency, the distinct affordances of modes, each with epistemological and/or ontological commitments. There is no possibility of a “perfect translation” across modes. Transposition and reconstitution of meaning across modes produces different “takes” on the world. To recapitulate: the transposition of the potentials for creating meaning enables meaning to be reconstituted in the new site. Meanings made in any one (social) site are constantly and frequently remade in different sites, in different social and semiotic environments. The sociologist Basil Bernstein used the sociological term re-­contextualisation when, for example, a professional practice involved in the making of an object – a wooden table, let’s say – is re-­contextualised as part of the school curriculum subject woodwork (Bernstein 1996). Re-­contextualisation is an entirely usual and unremarked, ubiquitous practice, in all of social life, in all kinds of social activity. In the classroom, where meaning is – or ought to be – the issue, the sociological practice becomes the semiotic matter of transposition of meaning. When a professional practice is re-­contextualised, its corresponding configuration of a multimodal

Transposing meaning  39

ensemble is reconstituted, and the resources for meaning-­making are transposed. Not only forms of speech or writing are changed, but many integral practices and objects are changed, as are their inter-­relations. In a not-­too-­distant past, when the assumption had been that what mattered was to adjust, adapt, and translate “the language” used in the originating site into “the language” of the target site, not all that much seemed at issue. There was a question of students being competent in the linguistic forms of the originating site and learning – gaining competence in – the usually different linguistic forms of the target site. This was understood to be a problem that had to be dealt with; the solutions differed in line with degrees and kinds of power. It pays to pause briefly over this example of the re-­positioning of a practice and the transpositioning and new constitution of meaning. In this instance – the professional practice of carpentry becoming the school curriculum subject of woodwork – the forms of speech in the originating site, the sounds, the words – the syntactic forms let’s say – will differ in some ways from those in the target site.Yet sounds remain sounds, broadly, even if not entirely the same; so do words, and so do – by and large – grammar and syntax. That is, the traditional notion of translation worked on the basis that the constituting entities of the source and the target language were broadly the same. Translation – as the transpositioning of meaning resources and the reconstitution of meaning – also applies to larger textual entities, whether these are treated as genres or as other larger text categories such as “the novel,” “the website,” “the opera,” or are seen as quite different kinds of categories, particularly across different societies. On this issue, see Kaindl’s chapter in this volume (Chapter 2).

7. The transposition of the “undocumented” and of “tacit” meaning A third-­year medical student and a surgeon are standing at an operating table. They are about to start operating on a small lump on the patient’s abdomen. As the patient is lying flat on his back, the lump is not visible. The operating light is focused on the patient’s navel. Before the surgeon makes the first incision, she points with her left hand to where the (presently invisible) lump that they will operate on is located and asks the student if he wants to have “a feel of that.” The medical student replies “yeah,” lightly touching at three different points around the focal area with a swab in his left hand. He then “feels” superficially with his right hand. He holds his hand flat, putting gentle pressure on various points with the tips of his fingers, covering an area of about three inches below the navel. He also makes a sweeping movement between two pressure points, as shown in Figure 1.4a. The surgeon then joins him in “feeling,” using her left hand. Her hand is slightly tilted, and she creates more pressure with the tip of her fingers, pushing deeper into the skin of the abdomen below the navel, as shown in Figure 1.4b. The pressure points mark out and make visible the circumference of the lump. This is then followed by a grasping action by the surgeon involving her middle finger and her thumb, which lasts for a couple of seconds.

FIGURE 1.4a 

Medical student touches patient

FIGURE 1.4b 

Surgeon touches patient

Transposing meaning  41

The surgeon’s action – her touch – is different from the student’s. The surgeon’s touch is more specific, deeper, firmer, involving (the tip of) a flat, angled hand, as well as a grasping action; the student’s touch is less focussed, more superficial, and involves (the tip of) his flat hand only. The different characteristics of their actions signify different engagement with the “lump” – the world at issue here – and, in that, distinctly different interest and knowing. The surgeon’s actions are designed to plan the incision: where to place the first cut with her scalpel. Her actions are guided by prior experience. The student’s actions are designed to feel the lump; he has had no prior experience to draw on. The surgeon knows, broadly, what to expect; the student doesn’t. Thus, the actions of surgeon and medical student point to different resources, to different means of (embodied) knowing; they demonstrate different histories of prior experience and knowing. Their actions, as signifiers, are apt to carry the meanings of each: the signs which each makes are motivated. At the same time, the actions of both lead to and constitute an extension of experience.With the making of their actions/signs, their semiotic work – much as a child’s making marks on paper – they expand their resources for making, transposing, and constituting meaning: in this case, resources for “reading” lumps of a certain kind and the specifics of the lump of this patient. In this action and interaction, in the demonstration of experience and inexperience, both have learned about the other sign-­maker: the medical student has interpreted how the surgeon touches, and the surgeon has interpreted how the student touches (Bezemer and Kress 2014). Transposed to a social semiotic multimodal account, we can say that every sign made here is new, is an “innovation”; its making is an act of “creativity.” If the example of the car is an instance of the constitution of meaning, here in the example of touch, we have an interaction in which each of the two participants interprets the other’s actions-­as-­signs. It is an example of the transposition and reconstitution of meaning by a very different route from those mentioned so far. It takes place in the student’s interpretation and his tentative “simulation” of the actions/signs of the initiating participant. By the end of this brief segment, the student’s touch has become different. The signs made by each participant are interpreted by the other, on the basis of the existing experiential/semiotic resources of each of the two participants. The point to draw out is that the transposition and reconstitution of meaning – of all sign-­and meaning-­making – is always based on the existing resources of each participant. It does not depend on or apply merely to the canonical means of documentation: of image, writing, speech. In this process the resources of each participant change. The meaning constituted by one participant is reconstituted in the interpretation by the other. In each case this happens on the basis of existing – and constantly changing – resources, which shape engagement, attention, selection, transposition, and reconstitution of meaning. The effects of the interpretation of each other’s sign-­making (the constitution and reconstitution of meaning) have changed the “inner” resources of each participant. That change in resources will be evident in their subsequent actions and will shape all future actions. For instance, in subsequent actions, certain features of their

42  Gunther Kress

actions might be highlighted: a foregrounding through slow motion, say, or attention to the precise direction of a movement. With every sign made, the sign-­maker’s knowing is transformed; this applies to student and surgeon. Neither could not learn from touching the patient or from observing the actions of the other. If translation is the process of making knowledge and understanding available and accessible to someone else by means of semiotic resources which that person does not yet possess (fully), then the process of demonstration here might well count as an instance of translation – of the transposition of knowledge.We can say that the reconstitution of meaning is an act of translation; or, that translation is the reconstitution of meaning for specific others; or that translation is the rhetorical act of the reconstitution of meaning for specific others, with specific (rhetorical) aims in mind. In the case of touch – the case of the “lump” – the question of the formal documentation of the entities involved in the reconstitution of meaning did not arise. For both participants the entities were those of the mode of touch: even though not named nor formalised, beyond unspoken suggestions/indications about intensity, angle, direction, density. The entities were observable but were not given other formal realisation or names. Yet the idea of the reconstitution of meaning exists there as much as in the reconstitution from one stretch of speech to another, or across modes. If there was to be a need to include a segment on touch in a textbook or on an online resource for beginning clinicians, then the question of the transcoding resource would arise and the question as to what would be apt modes and apt metaphors. As a last example here is an investigation of “tacit meaning”: it is another example from schooling, an ephemeral event in “early years.” In a social semiotic perspective, a chasing game, played by say four-­year-­old boys, is an ephemeral event. Whether it is going to be considered to be significant will rely on the semiotic work done and its persuasiveness. The event is deliberate, significant to its participants, and with significant social and emotional consequences inside this “early years” setting and beyond. Whether it is worthy of attention as an instance of meaning-­ making, such as drawing, writing, or making three-­dimensional models are seen to be, depends on whether semiotic means can be found to demonstrate its pedagogical significance. It is, in that sense, a question of translation: the documentation of meaning and the transposition of that meaning into a pedagogical environment. In preschool settings – and not only there – meanings made beyond “canonical means” are often overlooked; or, as in this case, disregarded as trivial: “boys just running around.” A major problem is methodological: how can we document, “fix” (Kress 2000) the “goings on,” to provide useable, reliable documentation, a record, of a running game, which will make it possible to analyse the meanings made and thereby widening the scope of what can get recognised as meaningful in young children’s meaning-­ making (see also Bezemer and Mavers 2011; Cowan and Kress 2017). In playing this game, children use a range of “ephemeral” modes, such as gesture, gaze, speech, features of movement; that is, all semiotic modes which leave no material traces.Yet, as potential data, these can constitute a rich set of materials. The meanings made in the game belong to, or in, the community in which the game exists. The multimodal

Transposing meaning  43

character of the game rules out attempts at a traditional written “transcript,” even if that were an apt means of (re-­)constituting its meaning. A written vignette is not a reconstitution of meaning in the sense so far discussed. A vignette cannot depict the children’s many rapid, changing movements and their complex use of the space: a vignette is a commentary rather than a reconstitution of the meaning. The problem is to identify, to document the meanings, the signifieds, and to decide what material means can serve aptly as signifiers. Some of the material features here which are giving material realisation to social signifieds are speed and control of speed, direction and control of direction, distance, and bodily position. Socially, the game is about a hierarchy to be established via the social signifieds; semiotically, the material signifiers are spatial and temporal in various ways. In her study, the researcher Kate Cowan chose a map-­like “genre” of layout as an apt frame – the “ground” – for documenting the salient features of meaning of a game of chase: making use of spatial-­visual layout and colour to give approximations of the modes used by the children. Figure 1.5 represents a 30-­second extract of George and Billy’s game of chase, offering a bird’s-­eye view of part of the outdoor play area. Lines represent the movement of each child; the arrows along the lines indicate the direction of their running; the arrows are positioned at approximate one-­second intervals to show when the children were moving slowly (the arrows being closer together) or at speed (the arrows being further apart). Whilst this can represent distance, direction, and speed, it less adequately represents sequentiality, and so multiple “fixings” were produced for the different stages of the game. As stillness was of significance as well as action, pauses were incorporated into the design using a circle with a

FIGURE 1.5 

“Fixing” the ephemeral: documenting meaning in a game of “chase”

44  Gunther Kress

number denoting the time (in seconds) that the children were still. The children’s talk was incorporated by locating speech bubbles at the relevant points along their movement “path.” The notational resources used were carefully selected on the basis of their aptness as means for documenting the most salient features of the episode. The task undertaken by the researcher here is to construct an apt means of documenting the social and semiotic event that had taken place. It is clear that the tools for documenting the meanings – the signifiers – that had been made are metaphors: for instance, distance covered as a means of indicating speed. We might note as a general point that the resources which might serve for documenting the reconstitution and transposition of meaning, in this instance, are material metaphors of the signifieds which are at issue. That formulation, however, also applies to the (various) commonly used practices of “transcription,” for instance, where sounds of speech (treated as phonemes) are represented by letters of the alphabet (or various often complex supplements to the alphabet, such as dots, dashes, and lines of various kinds to indicate further aspects of spoken interaction). Or in other forms of dealing with speech, where “words” (as signifiers) are taken as the basic units for documenting the reconstituting of meaning. The “documentary fixing” (Kress 2000) of ephemeral aspects of meanings of the game did foreground and make visible certain features that had not been immediately obvious from the original video recording. The shift of perspective, as much as the act of mapping itself, draws attention to patterns and points of interest, enabling scrutiny of movement and use of space as a crucial aspect of the play. In this way, the process of notation supports certain noticings. The mapping highlights how the rules of the play are communicated through subtle combination of movements, including decreasing speed, changing direction, and keeping other participants at a distance through an outstretched arm gesture. The play is successfully “wound up” by one participant before they discuss the reason for stopping, which ensures avoiding being caught and surrendering. In this way, the chase is paused and suspended, subtly communicating the message of “truce” through multiple ephemeral modes. In this example the reconstitution of meaning happens across two communities: that of the children at play in the formal setting of “early years” schooling and that of the researchers puzzling about the meanings of the children’s actions, seen as semiotic work. It is an instance of “noticing,” documenting, and transposing meaning.

8. “Naming” The world of meaning and of the making of meaning is in the process of a profound change: a paradigm change from a focus on language to a focus on meaning; a change from a focus on one to many means of making meanings evident; and a shift in disciplines from linguistics to semiotics. If that is the case, then the terms which named the former world cannot deal with the new conditions and practices. Many of the disciplines, still in use, were developed to deal with problems in a social world that was entirely differently conceived: “stability” might be a short-­hand means to

Transposing meaning  45

refer to that; while “provisionality,” “fragmentation,” and “diversity” might be terms that are apt for the present time. A central part of notion of stability was the role of language, as one central guarantor of meaning and of safe exchanges of meaning. For the issue at hand here – translation – the change away from “stability,” coupled with a concurrent move from an assumption of one means to many means of making meaning evident, is having far-­reaching consequences wherever meaning is the issue: in both theoretical and practical terms. The names that served the former order entrenched its theoretical notions, propping up the edifice of stability with great effect. The need is for names – a coherent terminology – which captures and aptly expresses the theory and its elements, practices, and consequences. Social semiotics, the theory proposed here to deal with “the issue at hand,” has “process” in the form of ceaseless semiosis at its core. The three central issues in sign-­ making – interest, criteriality, and the newly made motivated sign – are crucial in naming and in doing away with the misleading view of the “transport of meaning.” As the choice of an apt signifier for a signified is based on the interest of the sign-­ maker, each sign that has been made leads to a transformation of the signifier that was used. From that perspective there is no possibility of stable meanings. As the signified is shaped by the always varying conditions of each prompt and of what is seen as criterial at that point, there is no stability of the signified. The condition of the always new making of the sign, both in the initial sign-­maker’s production and in the interpretation by the addressee, means that the idea of stability in semiosis is unfounded and a problematic distraction. However, as the existence of a plurality of modes available to each social group is now a commonplace, it means that the practices and processes of changes in meaning have become everyday issues: whether banal or profound, within and beyond social groups. Apt naming has become essential and urgent. Different criteria and principles play their part in naming. To start with the term proposed in this chapter, transposition: it responds to two aspects of the contemporary social and semiotic world. One, meaning is always made in social interaction, in a social environment. There is a proliferation and diversity of social sites. “Position” is crucial in sign-­and meaning-­making. Prompts (as the material means of making meaning evident) are constructed initially in a specific social site and are interpreted in a different site. These are social aspects of “position,” and the morpheme “trans-­” signifies a change in these aspect. Two, there is a recognition of the multiplicity of modes – resources for making meaning evident – each with specific affordances and with accompanying epistemological and/or ontological commitments. The choice of modes represents a semiotic positioning, arrived at by the communicational requirements both of the initial maker of a sign or sign-­complex and the interpretative semiotic work of the person who engages with the prompt. Meaning is made in each position, both by the initial maker (of the text as message) and by the interpreter of the message. For both there is the sequence of interest → attention to part of the world in focus → motivation for (principled) → selection of that which is (either) to be → produced or interpreted. The difference between the two positions is that the initial maker sets the frame and the contents

46  Gunther Kress

of the frame, whereas the interpreter selects from the prompt that which is to be framed and what from within that frame is to be interpreted. That is, the initial maker has a wider range of choice: she or he sets the agenda, so to speak. Transposition names the fact of a change of “position” – both of the social and the semiotic positions. It indicates that the site where meaning is made has been changed; there has been a change in that. Transduction names the process of change of mode; though transduction does not indicate which modes have been involved. Transduction names a process in which ontological change takes place. That is, the source and the target mode have entities of a different kind: whether in terms of the logics (of temporality or spatiality) or of “substance”: e.g., the syntactic relation of two or more elements versus the spatial, locative relation of two or more elements (e.g., centrality, adjacency, marginality). Transcription, by contrast, names the manner and the means of the relation of two modes: the relation of speech (sound) and writing (script) in societies that use an alphabetic script. The alphabet is not an ideal resource for the transposition of meaning: the relation of phonemes (the theoretical generalised entities of speech) and graphemes/letters is always metaphoric and sustained by the power of convention. The term transcription is used elsewhere, as in the mode of music, in a manner analogous to the relations of speech and writing: i.e., the transcription of an orchestral piece to a piece for solo instrument. While transduction is used for transposition where two modes are involved (the entities of the one are unlike those of the other), transformation names a process of change within one mode and its elements. Transformation involves changes in ordering, e.g., of deletion and addition. It also involves changes from one textual – generic – entity to another; for instance, the change of a report to a narrative. Several of these distinct processes may occur in multimodal complexes; where one modal element may be transducted, another may be transformed, and yet others remain, in terms of mode. The social environment may be changed from one occasion to another, leading to transposition in addition to modal changes.

9. Conclusion On the one side, interlinguistic, intralinguistic and intersemiotic translation pointed out by Roman Jakobson (1971) are obviously describable on the basis of a single translation process model. On the other side, all types of communication in culture could be presented as the process of translation of texts (or fragments) into other texts. (Torop 2003, 271) What was the case in 2003 is clearly much more the case now, at a time when multimodality has become widely noticed and widely adopted, albeit in theoretically pretty well unconstrained ways.

Transposing meaning  47

Some things are clearer now. As I have tried to imply, the notion of “intersemiotic translation” finds no grounding in social semiotic theory: it assumes that all meanings made in one society are dealt within the one theory. The fact that meanings appear in many ways is dealt with by the semiotic category of mode: differently made evident in image or gesture or writing, but made evident in the categories of the one theory. My account is the merest sketch: I have not, for instance, concerned myself with larger semiotic (and social) entities such as “the novel,” “the film,” or “the opera.” The problem of naming is not restricted to the domain of translation: it affects all of the disciplines in the Humanities and Social Science, and no doubt beyond. One problem is the disappearance of formerly reasonably firm boundaries.When scholars from mass media studies discuss a notion such as “media” with scholars from literary or social or cultural studies, the debates tend to be rehearsals of “how we have always used that term.” I feel confident that the principles sketched here would – in more expanded and detailed form – provide resolutions to many of these questions. I remain puzzled by how one might translate semiotic entities such as cartoons produced over the last years by the English cartoonists Martin Rowson or Steve Bell. The social conditions in which they are produced would be the first point of investigations. One might find affinities with the cartoons, drawings, and etchings of William Hogarth or the paintings of Hieronymus Bosch. That would hint at certain regularities in the relation between the social and the semiotic. What could be adequate transpositional resources? In a different vein, there is the question of “What is and what is not translated? The libretto of the Mozart opera but not the score; the poems of the Schubert song cycle but not the music?” There are contemporary stagings, for example, of Wagner’s Ring as a feud in a capitalist industrial family, while the libretto or the musical score are left untouched. Helen Julia Minors reflects upon some of these issues at greater length in her chapter for this volume (Chapter 7). These, along with many other issues, will need to form the agenda of the next two decades or more. The question “what is translation” has, if any answer, an institutional one. The real issue looming is not technology but the future shape of what we call “the social.”

Acknowledgements I wish to thank the organisers of the CRASSH conference (Cambridge, July 2018) for inviting me to speak and the reviewers of my draft paper for helpfully difficult comments and questions. My colleagues Kate Cowan (who gave me permission to use Figure 1.5) and Jeff Bezemer (who gave me permission to use Figures 1.4a and 1.4b) have been constantly generous in allowing me to share aspects of their grappling with their research. Last but not least, thanks to Prue Cooper, the MC at the event I recount, for great kindness, generosity, and stimulation. I alone am

48  Gunther Kress

responsible for the awkwardness in attempting to express what is still not clearly worked out and trying to be clearer in getting a grasp on this topic.

References Bernstein, Basil. 1996. Pedagogy, Symbolic Control and Identity: Theory, Research, Critique. London: Taylor & Francis. Bezemer, Jeff, and Gunther R. Kress. 2014. “Touch: A Resource for Making Meaning.” Australian Journal of Language and Literacy 37 (2): 77–85. ———. 2016. Multimodality, Learning and Communication: A Social Semiotic Frame. London and New York: Routledge. Bezemer, Jeff, and Diane Mavers. 2011. “Multimodal Transcription as Academic Practice: A Social Semiotic Perspective.” International Journal of Social Research Methodology 14 (3): 191–207. Burn, Andrew. 2014. “The Kineikonic Mode: Towards a Multimodal Approach to Image-­ Media.” In The Routledge Handbook of Multimodal Analysis, edited by Carey Jewitt, 375–85. London and New York: Routledge. Cowan, Katharine, and Gunther R. Kress. 2017. “Documenting and Transferring Meaning in the Multimodal World: Reconsidering ‘Transcription’.” In Remixing Multiliteracies: Theory and Practice from New London to New Times, edited by Frank Serafini and Elisabeth Gee, 50–61. New York and London: Teachers College Press. Gibson, James J. 2014. The Ecological Approach to Visual Perception. London and New York: Routledge. Halliday, M. A. K. 1978. Language as a Social Semiotic. London: Edward Arnold. Hodge, Robert, and Gunther R. Kress.1988. Social Semiotics. Ithaka: Cornell University Press. Jakobson, Roman. 1971[1959]. “On Linguistic Aspects of Translation.” In On Translation, edited by Reuben A. Brower, 232–39. Cambridge, MA: Harvard University Press. Kress, Gunther R. 1997. Before Writing: Rethinking the Paths to Literacy. London and New York: Routledge. ———. 2000. “Text as the Punctuation of Semiosis: Pulling at Some of the Threads.” In Intertextuality and the Media: From Genre to Everyday Life, edited by Ulrike H. Meinhof and Jonathan Smith, 132–53. Manchester: Manchester University Press. ———. 2001. “You’ve just got to learn how to see: Curriculum subjects, young people, and schooled engagement with the world.” Linguistics and Education 11 (4): 401–15. ———. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. London and New York: Routledge. Kress, Gunther R., Carey Jewitt, Jon Ogborn, and Charalampos Tsatsarelis. 2001. Multimodal Teaching and Learning:The Rhetorics of the Science Classroom. London: Bloomsbury. Torop, Peeter. 2003. “Intersemiosis and Intersemiotic Translation.” In Translation Translation, edited by Susan Petrilli, 271–82. Amsterdam and New York: Rodopi. van Leeuwen, Theo. 2005. Introducing Social Semiotics: An Introductory Text. London and New York: Routledge.

2 A THEORETICAL FRAMEWORK FOR A MULTIMODAL CONCEPTION OF TRANSLATION Klaus Kaindl

1. Introduction Multimodality has become a widely used buzzword in many disciplines. As discussed in the Introduction, translation studies reacted to this trend – albeit hesitantly, at first – not least due to the booming field of audiovisual translation, which evolved from a relatively neglected subject area to a central research focus of the discipline (see the chapter by Luis Pérez-­González in this volume). However, this may at times give the impression that multimodality serves merely as a cosmetic means of concealing a continued focus on linguistic aspects in translation studies. In this regard, translation studies are still a long way from a semiotisation, which, according to Berressem (2004, 219), is a general trend in science.1 The aim of this chapter, firstly, is to situate multimodality in the semiotic research tradition with regard to its novelty value. Secondly, it will outline the efforts of translation studies to develop a multimodal understanding of text and, thus, also of the discipline. Based on this, the added value of a multimodal approach to translation studies and its practical implications for the definition of translation will be discussed. Finally, a multimodal approach to translation analysis will be presented through the examination of a rock ‘n’ roll song by Elvis Presley.

2. Multimodality as academic rock ‘n’ roll Multimodality appears to have replaced outdated semiotics and is often regarded as a leap forward compared to old sign-­theoretical approaches that emerged from semiotics. In translation studies, too, the mode concept seems to have forced out the semiotic sign concept. Since the semiotic dimension, which was introduced into translation studies by Jakobson (1959), was to play a central role, and an awareness of former theoretical and conceptual contexts of the scholarly debate is key to

50  Klaus Kaindl

meaningful progress, the consistencies between semiotics and multimodal discourse as well as the historical development, which is largely neglected by Kress and van Leeuwen (2001), will be outlined first. Progress in science can be determined by various criteria: it can indicate an increase in scientific knowledge, provide solutions to problems, or give us a new view of a section of reality, e.g., an object of research.2 However, some theorists are also sceptical about such a linear progression of scientific advancement. For example, Thomas Kuhn, in his work The Structure of Scientific Revolutions (1970), questions the view that scientific progress can be described as a cumulative gain in knowledge. To clarify the question whether the concept of multimodality has a novelty value, and, if so, what it consists of, I will adopt a quite unorthodox scientific-­ theoretical analogy and use the musical genre of rock ‘n’ roll as illustrative material for a multimodal translation analysis in the further course of this chapter. Rock ‘n’ roll was, as will be described in more detail later, a genre that shook the conservative 1950s. It was perceived as something completely new and revolutionary. De facto, however, it was not a new form of music, but merely a combination of different musical styles – elements of “white” country music, at least in rockabilly style, formal and melodic characteristics of “black” rhythm and blues as well as balladesque elements. Therefore, the novelty did not consist of some creation ex nihilo, but of a combination and re-­evaluation of individual, already existing musical components. The concept of multimodality presents a somewhat similar case. Since its introduction by Kress and van Leeuwen (2001), multimodality has been regarded as a new way of understanding the semiotic complexity of texts. However, the individual components that make up multimodality are not entirely new. In a way, Kress and van Leeuwen have created academic rock ‘n’ roll: they combined and reassessed already existing knowledge, primarily from semiotic studies and intermediality research. The following brief outline of the history of multimodality is not intended to diminish the merits of Kress and van Leeuwen, but to acknowledge the previous achievements of the semiotic tradition and to clarify what the contribution of the multimodal approach ultimately consists of. In a sense, such a critical appraisal follows a fundamental principle of semiotics, which, according to Julia Kristeva (1968/1977, 38), lies in viewing one’s own research subject not as a static truth, but rather as a never-­ending research process that must be constantly reflected upon and critically re-­evaluated. The cornerstones presented by Kress and van Leeuwen in their book Multimodal Discourse (2001) – the social contextualisation and social use of modalities, modes as a result of cultural processes, the functional entanglement of modes, the close relationship between mode and medium – are all aspects that were already discussed in semiotics in the 1970s, albeit under the heading of multimediality. This can be traced back to Charles Sanders Peirce, who pointed out the limitations of sign theory much earlier and suggested to place it in a medial context: “All my notions are too narrow. Instead of ‘Signs’, ought I not say Medium?” (Peirce 1906, cited in Walther 1997, 79).

A multimodal conception of translation  51

When Roland Posner (1976a, 39) noted that the theoretical prerequisites for multimedial (in today’s sense of multimodal) communication were missing, a number of semioticians took it upon themselves to develop building blocks for such a theory in the following years. Posner himself provided an outline of the key issues for semiotic studies, which he saw primarily in developing sign systems, which are not viewed as neutral, static units, but rather as a means of communication that perform various functions (cf. 1976b, 2). Hence, Posner prefers to use the term “Zeichenverhalten” (“behaviour of signs”) (1976a, 25) rather than “signs,” in order to emphasise their dynamic and changeable nature. According to Posner, signs do not only change synchronously – with regard to the purpose of communication and the respective context in which they are used – but also diachronically through a change of usage over time. Thus, if the multimodal approach explicitly distances itself from a structuralist view of the sign concept (e.g., Leeuwen 2005, 47–48), it only follows those trends in semiotic studies that already departed from a static code concept and a representationalist conception of signs in the 1970s.3 For Posner, signs are also always bound by their “relationship to the medium.” According to him, the associated sensory modalities and the communication channels used cannot and should not be examined separately. It is only through the “mutual relations” (1976a, 25, my translation) between sign and medium that the sign behaviour can be adequately analysed. The development of a corresponding terminology for exploring the functions of signs in relation to their media remained a desideratum for Posner: “Moreover, the relationship between code, channel and modality is still unclear” (1976a, 39, my translation). Ernest Hess-­Lüttich, among others, took up Posner’s suggestions and designed a research programme in 1978, which incorporated what would later become the central components of Kress and van Leeuwen’s multimodal discourse. His aim was to create a research framework that goes beyond semiotic analyses in various fields, such as film studies, musicology, art and literature studies, sociology, and psychology, and would enable the development of an interdisciplinary notation and analysis system. In that framework, multimediality incorporates the medium as a textual means in the broadest sense, social processes, sign systems, their technical distribution, and institutional systems for the realisation and organisation of medial products. Hess-­Lüttich emphasises the “communicative-­functional conditions of social interaction” (1978, 38, my translation) as being the decisive point of reference for the analysis of how the various semiotic resources and medial realisations are connected. Similar to Kress and van Leeuwen, he views the functional entanglement of semiotic resources and their embedding in social action contexts as the central starting point for research into multimedial or multimodal communication.4 The fact that these functional relationships are not static but dynamic as well as culturally specific, as Kress and van Leeuwen emphasise, was also recognised and incorporated very early in semiotic studies. Kloepfer, for example, in his analyses of text-­image relationships in various medial contexts, shows that these are “dynamic, polyfunctional and also serve to change the underlying codes” (1976, 42, my translation). The same also applies to the role of culture in shaping the meaning

52  Klaus Kaindl

of semiotic resources, which van Leeuwen seems to miss in traditional semiotic approaches to sign systems (cf. 2005, 160). Umberto Eco noted as early as 1972 that the meaning of a sign is not derived from its relationship to the object, but is the result of perceptive and cognitive processes, which, in turn, are influenced by culture. Eco illustrates this with the example of a zebra, which in our culture is primarily identified by its stripes, whereas in another culture, where it is more important to distinguish a zebra from a similarly striped hyena, it is identified by its shape, which for us resembles that of a donkey (cf. 1972, 202). The embedding of the mode concept into a social-­communicative context, as suggested by Kress and van Leeuwen, was thus already postulated and developed in semiotics in the 1970s, but perhaps not consistently applied. Multimodality research also shares its goals with semiotic studies, which, according to Jewitt (2014a, 17–20), consist of describing semiotic resources and investigating their relations and the impact of media. Nevertheless, multimodality offers a novel approach due to the combination of concepts, not so much on account of the terminology,5 but because of the differentiation of its modal components. The four dimensions of mode – discourse, design, production, and distribution – sharpen our view of the different dimensions of signs and modes, which enables us to grasp the functional interaction between the semiotic resources and their medial realisations more precisely. At the same time, consistent socio-­cultural embedding brings into focus not only the dynamics of how meaning is constructed, but also the fact that such changes do not occur arbitrarily but according to socially established rules. Thus, a frequently neglected factor in semiotics takes centre stage: power in Foucault’s discursive-­ political sense as an integral component of signs or modes. In contrast to semiotic approaches, the question of how mode and social identity are connected can also be answered with the sociological inventory of multimodal discourse. Therefore, the emphasis on the sociological, functional, and medial aspects of the multimodal approach makes it so interesting for many disciplines, including translation studies. As previously discussed, these factors were already present in applied semiotics in the 1970s, but translation studies was still far too strongly tied to the linguistic paradigm at that time, which resulted in a merely rudimentary integration of semiotic considerations into the scholarly debate. While the text concept remained static for a long time, particularly in translation studies (cf. Kaindl 2013, 259–60), the multimodal approach captures its functional dynamics. These dynamics result from the cultural and sociological variables and the respective forms of medial realisation. The fact that the shift in translation studies towards culturally sensitive, sociologically motivated, and medially conscious research coincided with the development of the multimodal approach may be one of the reasons why translation studies is increasingly incorporating it into its scholarly investigations. The paths to scientific progress are manifold and not always straightforward. The present case illustrates that progress does not necessarily happen cumulatively by building on what already exists. Rather, Karl Popper’s view of progress applies here, which – in reference to Darwin – occurs as a result of natural selection (cf. 1998, 273). Kress and van Leeuwen provide a modern take on a well-­known

A multimodal conception of translation  53

phenomenon by combining and re-­evaluating previously researched aspects, which shows that a scientific approach is ultimately more than the sum of its parts. As a next step, I will critically examine the possible consequences of this holistic view for translation research.

3. On the (bumpy) road to multimodal translation studies Translation is – and this it probably shares with mode – a colourful phenomenon, which appears to elude clear, universally valid definitions. Definitions, however, are often considered as a prerequisite for scholarly research or, to put it another way, a necessary evil. An evil, because definitions – at least of cultural and social phenomena – are not neutral, innocent acts but attempts to “discipline” an object of research, which is always guided by particular interests. In the case of translation, these disciplinary measures are probably also one of the reasons why translation studies struggle with multimodality. Its monomodal orientation can be explained with the emergence of the discipline after World War II. The initial focus of machine translation in the 1950s was on language systems and the question of how to create the greatest possible invariance between word and sentence units of two languages (cf. e.g., Bar-­Hillel 1960). Roman Jakobson’s realisation from 1959 that we communicate not only with language, but also use other communicative means to make ourselves understood, went largely unnoticed at the time and only found resonance in translation studies at a much later point. In particular, the integration of audiovisual translation into translation studies, which began in the 1990s, has made a significant contribution (e.g., Delabastita 1989; Gottlieb 1994). As the summary in the Introduction indicates, translation studies initially remained a purely language-­centred discipline that opened up to text linguistics in the 1970s and, thus, increasingly included pragmatic aspects in its research. The sender and his/her intention, the recipient and his/her level of knowledge and understanding, as well as text types and their conventions, now dominated the discussion. Rigid demands for equivalence (e.g., Catford 1965; Kade 1968), which aimed at producing the greatest likeness of a linguistic unit from one language to another, were now bent to ensure that the effect of source and target text would be as similar as possible on the recipients (e.g., Nida and Taber 1969; Reiß 1971). However, the fact that in text linguistics and semiotically oriented linguistics, texts were seen not only as linguistic but also as non-­linguistic units was – as we will see later – only hesitantly integrated in translation studies.6 For a long time, translation studies remained a discipline whose focus was on the transfer of meaning from one language to another.7 When the so-­called success story of translation studies (Bassnett and Lefevere 1990, ix) began in the 1980s and 1990s, this was primarily due to an expansion of research interests to include the cultural and sociological dimensions of translation. Translation was no longer merely a transfer of linguistic meaning, but a multi-­layered transfer of culture, ideology, power, gender, race, etc.; and the focus of translation research was no longer on the source text as the main point of reference, but rather on its functioning in

54  Klaus Kaindl

the target culture. These new perspectives, be they the descriptive, culture-­sensitive approaches of Toury (1995) and Lefevere (1992), or the functional, goal-­oriented approaches of Vermeer (1986) and Holz-­Mänttäri (1984), generated an awareness for previously largely neglected areas such as audiovisual translation, the translation of theatre plays, illustrated children’s books, and, at the turn of the millennium, websites, video games, etc. For a long time, all these texts were excluded from the translation studies research canon, as they could not be examined using source-­text focused equivalence criteria. Moreover, the monomodal view of texts did not allow for a high proportion of non-­linguistic text components or for medial realisation to influence the composition of texts. Nevertheless, the verbal fixation continues to have a firm grip on translation studies to this day, and the expansion of translation theory and analysis tools to other modes and media is taking place rather hesitantly and in an unfocused manner. Therefore, building on existing approaches, the following section will present basic principles that delve deeper into the multimodal reality of communication and hence also translation.

3.1.  Building blocks of a multimodal theory of translation The elements of a multimodal communication theory, as defined by Kress and van Leeuwen, represent an important basis for exploring multimodality in translation studies and can be related to communication-­sensitive and culture-­sensitive translation theories. A theoretical approach in translation studies, which has numerous relations to Kress and van Leeuwen, is Holz-­Mänttäri’s translation theory (1984), based on action theory. Similar to the role of human beings, “their social agency” presenting “a criterial aspect” in multimodal theory (Kress and Jewitt 2003, 9), Holz-­ Mänttäri regarded translation as an activity where different actors participate in developing a text. Holz-­Mänttäri emphasised – similar to Kress and van Leeuwen – the design character of translation. Translation cannot be reduced to the transfer of linguistic meaning, but it is designing texts across cultural barriers. However, the aim of a translator as a text designer is not to understand the text himself/herself, but to produce texts for the needs of somebody else (Holz-­Mänttäri 1993, 303). The design of texts across language and cultural barriers needs a specification for production; this is negotiated in the interaction of different actants who act as a part of a social complex of actions. Now, the source text is not the determining factor anymore; it has been replaced by the function of the translation in the target culture, which depends on the actual context of use as well as the expectations and the level of knowledge of the target audience. Translators are normally specialised in the transfer of verbal texts, so the multimodal design of texts often requires them to work with other experts like photographers, composers, graphic designers, etc. Concerning the production of design texts, Holz-­Mänttäri explicitly referred to their multimodal character and called them message conveyor compounds (“Botschaftsträger-­im-­Verbund” 1984, 76). The compound character of the different modes exactly corresponds to the

A multimodal conception of translation  55

functional relation, as Kress and van Leeuwen pointed out. Thus, in a technical manual, for example, the visual mode as well as the linguistic mode can take the explanation of the operating steps.Whether the communicative function is fulfilled by images or by linguistic explanations is, on the one hand, culture-­specific, and on the other hand, depends on the production context. Kress and van Leeuwen as well as Holz-­Mänttäri clearly pointed out that the individual steps of text production – from discourse and design to production and reception – determine which modes are used in which combination to achieve a communicative aim. While Kress and van Leeuwen pointed out the characteristics of multimodality in their theory with the transcultural aspect hardly playing a role, Holz-­Mänttäri above all investigated the steps of actions which are relevant for producing multimodal texts across language and cultural barriers. Thus, both theoretical approaches can additionally be related to each other for perceiving multimodality in translation studies. Translation studies, with its focus on transfer processes of modes across modal and cultural boundaries, can help identify the specifics of modes and their functions; and Kress’s chapter for this volume (Chapter 1) sketches a multimodal framework for this kind of analysis. A transfer is only possible if there are enough similarities between modes for them to be comparable and, thus, transferable, and if there is a sufficient difference that makes the transfer necessary. Therefore, translation gives us an insight into what modes have in common and what differentiates them. The contribution of a theory of multimodal communication for translation studies is the specification of the modality notion, which Holz-­Mänttäri mentioned, but did not elaborate comprehensively. In addition, it can end the conceptual confusion that prevails in this area of translation studies. The varied terminology that has emerged since Katharina Reiß’s introduction of the subsidiary, then audio-­ medial, and finally multimedial text has prevented a coordinated debate in translation studies based on a common definition.8 A multimodal understanding of translation requires that not only language and its functions/functioning, but also other central modalities such as music and image as well as the associated submodalities9 must be examined with regard to their functions and cultural specificity – and, in their different ways, the chapters in this volume by Elisabetta Adami and Sara Ramos Pinto (Chapter 3), Matthew Reynolds (Chapter 5), Marcus Tomalin (Chapter 6), and Helen Julia Minors (Chapter 7) all explore various complexities that arise from this process of examination. However, the majority of translation research dealing with nonverbal modalities regard them as an obstacle to translation work, rather than an integrative component of a functional text. Despite the fact that recent scholarly works incorporate multimodality as a basic principle, Jewitt’s (2014b) differentiation of various multimodal approaches shows that the potential offered by multimodal theory has not been fully exploited to date. It is fair to say that the majority of translation research falls into the category of social semiotic modality. According to Jewitt, the focus here is on different cultural and social contexts in which multimodal meaning-­making takes place. Numerous

56  Klaus Kaindl

works (e.g., Gambier 2006; de Pedro Ricoy 2012; Jiménez Hurtado and Gallego 2013; Borodo 2015; Cross 2016; Oittinen, Ketola, and Garavini 2017) adopt the term mode merely as a substitute for the outdated sign concept and neglect the other dimensions associated with mode as well as the social conditionality of modes, which ultimately leads to a rather rudimentary, not to say deficient, reception of multimodal communication theory in translation studies. Despite the fact that Munday (2014) argued that translation analyses of advertising should be increasingly examined from a multimodal point of view, theoretic scholarly work in this area is still very scarce. Among the few works are Ketola (2016), who developed a framework for the cognitive dimension of multimodal translation; Lee (2012), who examined modes with regard to their technological interconnectedness; and Pérez-­ González (2014), who argued for an integration of multimodality into translation and interpreting studies beyond the semiotic nature of modes. The other two approaches identified by Jewitt can hardly be found in translation studies at all or only in homeopathic doses. Exceptions include Baldry and Taylor (2002), who employed the second approach, multimodal discourse analysis, to identify functional units for subtitling. The third approach, multimodal interaction analysis, which would be particularly useful for interpreting situations, has hardly received any attention either, with very few exceptions (e.g., Krystallidou 2016). One of the characteristics of Kress and van Leeuwen’s multimodality concept is that mode is seen not only with regard to its semiotic nature, but also as inseparably connected to the medium. This dimension is particularly central to translation studies. Media are relevant for translation studies in two respects. On the one hand, media, like modes, are not essentialistically predetermined entities but rather culturally constructed mediation devices that influence both the selection and the functioning of modes. On the other hand, transfers do not cross only mode boundaries but also media boundaries, and the associated processes still remain relatively unexplored in translation studies. Since McLuhan described media as “living vortices of power” (1950, vi), which shape and change our society, a lot has happened in research. His view of media as being “extensions of man,” as indicated by the subtitle of his book Understanding Media (1964), would probably be inappropriate from today’s gender perspective, and the equation of medium and technology also seems a little too narrow. However, the fact that media are more than just a means of conveying certain information and that they also shape communication is still valid today and has increasingly been put into focus by intermediality studies.10 Like mode, the term medium is used by different disciplines in various theoretical and practical contexts and is, therefore, defined and understood in a variety of ways. Its ubiquitous presence makes efforts for a uniformly binding definition seem futile. What is required, however, is a coherent conceptual clarification for the respective approach. Kress and van Leeuwen define media as “the material resources used in the production of semiotic products and events, including both the tools and the materials used” (2001, 22). However, a focus on the technical-­material dimension of media is

A multimodal conception of translation  57

too narrow, as Elleström (2010a, 5) notes. Furthermore, he identifies “basic media” and “qualifying aspects of media” as core facets of the media concept, which, according to him, are central to a substantiated scholarly discourse. The former defines the basic characteristics of the medium, and the latter includes the historic, social, cultural, aesthetic, and communicative dimensions of a medium. The media spectrum encompasses not only production and carrier media, such as musical instruments, pens and pencils, cameras or books, computers, newspapers, etc., but also art forms such as films, opera, theatre, etc. Media differ, according to Elleström (2017, 668–69), not only in their material but also sensorial and spatiotemporal qualities. It appears crucial to me that media not only have a material and social aspect but also a semiotic and cultural one. The material dimension of a medium, e.g., the page of a glossy brochure or of a trashy magazine, creates expectations in the recipient that can decisively influence the constitution of meaning. The spatiotemporal and thus also sensorial differences between the book medium and the theatre medium influence the choice, arrangement, and perception of the modes used. Moreover, Elleström (2010b, 26) points out that a medium is also closely connected to the genre.11 The medium of film consists of the genres comedy, thriller, science fiction, etc. If we assume that every text is already multimodal (cf. Iedema 2003, 40; Gambier 2006, 6, among others), then this also means that genres per se are multimodal. Derrida (1980, 65) already noted that every text is associated with one or more genres. Genres are made up of a number of conventions, which include topics, characters, character constellations, content, narration, as well as the discursive and formal use of modes. They determine what readers expect, how they approach and interpret the text. Like modes and media, they have constructive characteristics and are culture-­specific, or as Elleström (2010b, 26) puts it: “A genre cannot be circumscribed as an abstract entity without considering how both ‘form’ and ‘content’ are related to both aesthetic and social changes.” Mode, medium, and genre are three building blocks that form the basis for a translation-­theoretical approach that serves to overcome the language-­centredness of translation studies and to understand translation as a modal, medial and generic practice. It must be noted, however, that mode, medium and genre are part of a closely interwoven conceptual network (see also Ryan and Thon 2014, 9–10).12 Their separation may have heuristic value on a systemic level, but in translation practice, they have to be regarded as three parts of one whole. If we take the idea of mode as semiotic resource seriously, this does not only mean that modes have no pre-­given meaning, but concretise their semantic potential only in a given communicative context. Consequently, we cannot understand and analyse modes in isolation, but only in their interconnectedness with media and genre. In order to illustrate these complex relationships, I would like to use the German translation of the opera Carmen as an example.13 Shortly after its premiere in Paris in 1875 at the Opéra Comique, Carmen was performed at the Vienna Court Opera. The change of mediation context determined the adjustment of the genre

58  Klaus Kaindl

to which Carmen pertains: the opéra comique. This genre has specific modal characteristics, such as the distribution of spoken and sung passages. The musical mode is also characterised by specific compositional styles; similar to the sung mode, it comprises a number of typical elements, which are expressed as “détaché,” “trés léger,” or “très également et simplement,” and which can be traced back to the roots of the opéra comique in spoken theatre. Carmen was explicitly composed for the Opéra Comique, and thus the medial context, genre, and intermodal realisation were all part of one whole. The decision to stage the German-­language performance at the Vienna Court Opera had a significant impact on the genre as well as the modal realisation and intermodal relationships associated with it. The much larger dimensions of the stage created new semiotic potential, which in turn affected the use of extras, props, and stage decor as well as the verbal, vocal, and musical mode. This required a modification of the original genre, as the intimate setting of the opéra comique was not suitable for the Vienna Court Opera. To overcome this problem, the translator, Julius Hopp, modified his text to suit the conventions of the Romantic opera, which was reflected in the musical realisation and the relationship between spoken and sung modes. The orchestra was extended, the spoken passages were deleted, and the singing style was adapted to the conventions of the Romantic opera. This also led to dramatic changes in the realisation on stage: the choice of costumes, the number of extras, and thus also the representation of the characters had to be adapted according to the new genre and medial conditions. The medial context, the genre-­specific conventions, and the modal relationships influenced each other and turned a realistic chamber play into a bourgeois Romantic opera.

3.2.  A multimodal definition and taxonomy of translation If the research focus of translation studies shifts from the linguistic aspects of a text to texts in all their modal realisations, medial contexts, and generic forms, then the scope of research has to be expanded to an understanding of translation as a conventionalised cultural interaction in which a mediator transfers texts in terms of mode, medium, and genre across semiotic and cultural barriers for a new target audience.14 A text that serves as the basis for a translation consists of the combined use of different modalities; these are based on distinct discourses, and they are produced, distributed, and received on the basis of a specific design. Such a definition differs from language-­centred approaches in two respects. On the one hand, there is no fixed relation to the source text based on similarity, equality, or equivalence and, on the other hand, the role of language in translation does not take centre stage. This also means that it is no longer necessary to rename translation phenomena that deviate from these criteria. For example, translations that differ from their source texts are frequently labelled differently, e.g., editing, free rendering, rewriting, appropriation, or adaptation. The same applies to translations that transcend modal boundaries. Depending on the discipline, we can find a variety of different labels: in translation studies we have transmutation in reference to Jakobson (1959), para-­translation (Yuste Frías 2012), post-­translation (Gentzler

A multimodal conception of translation  59

2017) or, more generally, adaptation or version. Media research makes use of terms such as medial transposition, transmedialisation, intermedial transfer; while in multimodal communication theory we can find transmodal moment (Newfield 2014), transmodal operation (Wyatt-­Smith and Kimber 2009), or transmodal redesign (Mavers 2011).15 The term transduction introduced by Kress plays a central role in this regard. Originally, it was used to describe how students create meaning across different modes. It was linked to translation at a much later point (2010), and more recently it has also been used in translation research (e.g., Poulsen 2017). For the reasons mentioned already, we will not use the term proposed by Kress,16 as it would limit the understanding of translation applied in current translation-­ theoretical approaches. There are, however, links to the characteristics associated with transduction. For example, what Kress says about producers of multimodal texts also applies to translators: they “stretch, change, adapt and modify all of the elements used, all the time, and thereby change the whole set of representational resources with its internal relations” (Kress 2000, 155). Similar to action-­oriented, sociological, and descriptive translation studies, Kress also emphasises the action component and the social, cultural, and historical conditionality of transfers across modal boundaries: “An adequate theory of semiosis will be founded on a recognition of the ‘interested action’ of socially located, culturally and historically formed individuals, as the remakers, the transformers and the re-­shapers of the representational resources available to them” (2000, 155). Newfield notes from a multimodal viewpoint that The concept of the transmodal moment focuses attention on the relational aspect of the transmodal chain, on the way in which a modal shift impacts on meaning and on the way in which the links are connected or discontinuous with one another. (2014, 103) The aspect of relationality, which is changed by transfer, also provides a parallel to translation research. In translation, a multimodal whole is also transferred into another temporal, social, and cultural context. Assuming that the meaning of modes always depends on the specific context of use, such a transfer also means change, since the recipients and with them the social and cultural parameters change. Thus, every translation is inevitably subject to change. Fixed, rigid relational specifications, such as the relationship between source and target text, are just as obsolete as the idea that modes could pass through the process of translation without undergoing change. Even though translation studies are still occupied with the linguistic dimension as its prototypical core area, texts and their transfers should no longer primarily be examined based on language aspects, but through the lens of mode, medium, and genre. These should also form the basis for a taxonomy of translation. One of the first scholars to establish a classification of translation that went beyond language was Roman Jakobson (1959) with the famous triad of intralingual, interlingual,

60  Klaus Kaindl

and intersemiotic translation. However, he remains committed to the linguistic paradigm and only considers interlingual translation to be translation in the proper sense, while intersemiotic translation, which he also calls transmutation (1959, 233), represents a special case. Jakobson’s classification may have been influential, but it proved problematic due to its conceptual inconsistency, which has been repeatedly criticised. The term intersemiotic translation is unfortunate inasmuch as language is also a semiotic system and, thus, the translation between two language systems would also have to be regarded as intersemiotic translation. Toury (1994, 1114) avoids this problem by defining Jakobson’s interlingual translation as intrasemiotic translation, which can be divided into intrasystemic (i.e., intralingual) and intersystemic translation. For the latter, he explicitly only gives interlingual translation as an example, but other modalities would also be conceivable here. Toury’s definition of intersemiotic translation, which translates between different modes, is similar to Jakobson’s. In principle, however, semioticity does not seem to offer itself as a suitable basis for a taxonomy of translation, due to the all-­encompassing variety of meanings of the term. Therefore, it is preferable to differentiate among the categories mode, medium, and genre. Thus, at the modal level, a distinction would have to be made between intra-­and intermodal translation, which can take place intra-­or transculturally. While intramodal translation involves translating within a modality (verbal-­ verbal, image-­image, etc.), intermodal translation exceeds modality limits. Examples range from the translation of a verbal instruction manual into a pictorial representation to the translation of bible text into a comic. At the medial level, intramedial transfers refer to processes where the medium remains unchanged. However, this can also pose problems due to cultural differences. For example, US-­American and European music videos differ on a visual level due to different editing and image sequences. Intermedial translations refer to a change of medium, e.g., from a novel to a screenplay and subsequently to a film. Finally, at the generic level, translation can also take place intra-­and intergenerically. Similar to the medium, culture-­specific problems can also arise in intrageneric transfers, for example, when writing conventions differ in two languages: in French, the alexandrine verse form is conventionally used for drama, whereas German drama is not characterised by the use of verse metres. Or, in the case of the translation of an Argentine tango into a Finnish tango, the two have different musical conventions. An example of an intergeneric translation would be the transfer of a French opéra comique into a German Singspiel or – as will be shown later– of an American rock ‘n’ roll song into a German Schlager. It should be noted that mode, medium, and genre are inextricably linked and that a change in one dimension fundamentally affects the functional whole, which consists of all three parts. Thus, it is impossible to examine mode, medium, and genre independently from each other in the context of translation. Nevertheless, the differentiation makes sense for translation studies in that all three components can each pose their own translation problems. These range from modal

A multimodal conception of translation  61

communicative-­functional aspects through medial technical-­social implications to generic text-­conventional issues.

4. Mode, medium, and genre in the translation of rock ‘n’ roll If translational action is understood as being semiotic and translation as a multimodal practice, then a number of questions and challenges arise for theoretical reflection on and the practical application of translation: • • • • •

What are the (multi)modal culture-­specific textual conventions? What meaning(s) is activated by which modes? How can multimodal translation units be identified? What does a multimodal translation-­relevant text analysis look like? Which modes are changed in the translation process and how does this affect the functional multimodal whole?

I would like to answer at least some of these questions with the example of the song Hound Dog (written and composed by Jerry Leiber and Mike Stoller, and performed by Elvis Presley) and its translation into German.17 The modal, medial, and generic specifics and their functional interconnectedness serve as a starting point for the multimodal translation analysis.18 As mentioned earlier, the development of the genre rock ‘n’ roll was shaped by cultural and medial factors. The 1950s economic boom in the US also created new financial opportunities for a previously economically weak social group, teenagers. The political situation of the Cold War and the associated repressive climate in the US increased the need for distraction and entertainment, and the gradual awakening of self-­confidence among the African American population contributed to the spread and popularisation of black forms of music, which were also adopted by white music producers for economic reasons. Record production and radio stations and their programmes became more diverse, and the technological development of new carrier media – record, tape, and transistor radios – contributed to increased proliferation. As far as music and language are concerned, rock ‘n’ roll is a classic example of the combination of black and white music. It is influenced by (white) country music – at least in rockabilly style – as well as by (black) rhythm and blues. Rock ‘n’ roll exhibits numerous formal and melodic characteristics of R&B, but we can also find balladesque elements. As mentioned at the beginning of this article, the novelty lies in the combination and reassessment of the individual components.19 At the same time, popular music in the German-­speaking world was dominated by the Schlager genre. The 1950s Schlager was light, popular music with a rather conservative touch, which was mostly geared towards evoking blissful and harmonious feelings, and its sentimental-­romantic lyrics were free of sexual themes or themes that might contradict accepted forms of relationship at the time. The

62  Klaus Kaindl

music, however, was quite flexible and absorbed different musical styles, which – as we will also see in the following example – were adapted to culture-­specific medial mediation contexts.20 Hound Dog was originally written for the singer Big Mama Thornton as a rhythm and blues number and intergenerically translated for Elvis Presley as a rock ‘n’ roll song. It became a number-­one hit in 1956: You ain’t nothin’ but a hound dog, Cryin’ all the time. You ain’t nothin’ but a hound dog, Cryin’ all the time. Well, you ain’t never caught a rabbit, And you ain’t no friend of mine. When they said you was high classed, Well, that was just a lie. When they said you was high classed, Well, that was just a lie. You ain’t never caught a rabbit, And you ain’t no friend of mine. (Presley 2002/2003) The story being told in the lyrics is about a lazy, useless person who is not much better than a lazy dog. Considering the background of the original singer, the slang contraction “you ain’t” could also be seen as a reference to the Afro-­American origin of the song. In a sense, one of the essential characteristics of rock ‘n’ roll, namely the combination of black and white music, is transferred to the verbal and vocal mode here. Elvis’s vocal characteristics – the throaty, rough voice projected with high pressure, the blurred notes that start as low tonal sounds and the “smearing,” the intonational smearing of notes, which make up his unique style (cf. Middleton 1977/1992) – are in keeping with the rhythm and blues tradition. The pressure he puts on his voice corresponds to the verbalised disappointment and anger, which gives the song an overall aggressive tone. The musical submodes of the song can be differentiated into harmony, melody, and rhythm. All three factors are in keeping with the classic blues tradition. The harmonic elements express the tension that permeates the song in the verbal and vocal mode: this is achieved by contrasting the major harmonies, which form the basis of the composition, with the “blue notes” in the vocals, in other words, with minor thirds. Moreover, the breaking up of the major power chord of the bass as well as the the I-­IV-­I-­V-­IV-­I chord progression create a mix of major/minor colouring of the song, which is typical of blues. The rhythm is primarily determined by the dotted quarter notes of the bass, and the blues style is further emphasised by the “claps.” The melody also exhibits classic blues characteristics. Blue notes create a minor coloration of the tune, which in turn contrasts the major harmonies.

A multimodal conception of translation  63

As far as the medial dimension is concerned, the sound design depends not only on the arrangement but also on the sound technology used. The high-­energy sound is created with the help of a specific recording technique that amplifies the sound electronically, which makes it powerful and loud at maximum sound levels. Both the individual modes and the production media have a functional relationship with each other, which gives the song its specific coloration and mood. In the German version, changes were made at the generic, medial, and modal levels: Ja das ist der neue Rhythmus, [Yes, this is the new rhythm,] Das ist Rock and Roll [This is rock ‘n’ roll] Ja, da wo jeder mit muss, [Yes, everyone wants to do it] Das ist Rock and Roll. [This is rock ‘n’ roll] Ja der Rock ist der aller letzte Schrei. [Yes, rock is the latest craze.] Jeder ist dabei. [Everyone is doing it.] In Peru und in Chile, [In Peru and in Chile,] In Paris und Madrid [In Paris and Madrid] Tanzt man Rock ohne Pause, [Everyone is dancing rock without a break] Und wir tanzen mit. [And we dance along.] Rock and Roll morgens mittags abends [Rock ‘n’ roll in the morning, at noon and in the evening] Tanzen wir nur Rock and Roll. [We all dance the rock ‘n’ roll.] (Presley n.d.) The functional connection between the modes – language, music, and vocals – and the medial production context differ from the multimodal make-­up of the source text. Both the multimodal and medial elements serve to adapt the rock ‘n’ roll number to the Schlager genre. In the language mode, a socio-­cultural change occurs in comparison to the original. The foreign genre is explained verbally and presented as dance music, which also finds reflection in the musical mode: the melody and rhythm of the original, as well as its composition, are toned down with a view to presenting a decent and clean dance style that is appropriate for young adults. The vocal mode does not produce any pressure due to intonation, volume, and articulation, and it sounds rather soft and instrumental; in other words, it is just a tonal addition to the composition. The melody does not contain any “blue notes,” which leaves us without the “dirty” rhythm and blues sound. Instead, the melody is played in several voices and in thirds above the keynote. The harmony is largely kept in major key, creating a serene sound, and the rhythm is based on that of the big-­band revues of the 1950s. The instrumental part is put in the foreground and harmonises with the lyrics, which deal exclusively with the new dance rock ‘n’ roll. This adaption is further reflected in the recording technique, which does not use any electronic sound compression but reproduces the classic sound of Schlager music. Although the same instruments – electric guitar, contrabass, and drums, with

64  Klaus Kaindl

the addition of the saxophone and trumpet – are used, the technical media create a sound that is compatible with Schlager music. Overall, the individual modes are in harmony with each other: as opposed to creating modal tension, language, music, and vocals support each other in order to present a “foreign” genre in a familiar and, above all, harmless guise, i.e., in the form of Schlager music. To a certain extent, the functional connections have been reversed in the translation. However, the translation represents a coherent multimodal whole, whose changes can only be explained by the socio-­cultural background of the genre into which it was translated.

5. Conclusion At present, the interdisciplinary interaction between the multimodal communication approach and translation studies still seems to be quite one-­sided. While the former is hardly aware of translation studies, the latter has recognised multimodality as a relevant topic. This is evidenced, on the one hand, by entries in handbooks and encyclopaedias (e.g., Kaindl 2013; Pérez-­González 2014), and on the other hand, by the large number of individual studies dealing with multimodality – albeit mostly in a very selective manner. Conversely, when the topic of translation is touched upon in multimodality research, a very outdated, static concept of translation frequently serves as the basis of investigation, and recent theoretical work in translation studies is almost never used. This article hopefully has shed some light on the common ground between the two disciplines: both deal with texts from a medial, modal, and generic perspective; both regard transformation and relationality as essential characteristics of their research subject; and both consider their subject in a social and cultural context, rather than as an abstract and static entity. This seems to be a good basis for increased interdisciplinary networking in the future. Moreover, increased networking across all disciplines dealing with modal, medial, and generic transfer processes would be desirable. In addition to translation studies and multimodality studies, these include intermediality studies and adaptation studies. Translation studies has recently entered into a mutual dialogue with the latter in particular. While the first edition (2006) of Linda Hutcheon’s standard work A Theory of Adaptation still contains no mention of possible points of reference to translation studies, the new edition (2013) is at least aware of it. A dialogue between the two disciplines, as conducted by van Doorslaer and Raw (2016), in order to work out the similarities but also differences between the two disciplines, could also be an example of a future exchange between the two transfer disciplines. A dose of multimodal awareness would undoubtedly be beneficial for translation studies as well as translators. This means recognising other modes beyond language as relevant for translation and developing corresponding transfer skills. This does not mean that translators must be able to compose, draw, create graphics, and the like. However, they must be able to transfer the functional whole of a multimodal text in cooperation with other experts in accordance with the expectations and requirements resulting from the context of use and reception in the target culture.

A multimodal conception of translation  65

This requires not only classic translation skills, such as linguistic and cultural competence, specialist competence, research competence, and transfer skills, but also comprehensive multimodal awareness, which also includes media and genre awareness. In concrete terms, this means that translators must be able to classify semiotic resources, analyse their meaning and functioning, and understand their functional links with other modes. However, what Jäger called “transcriptive intelligence” (2002, 35), i.e., the innate ability to communicate and transpose content across mode and media boundaries, is not sufficient for professional translatorial work. A (professional) translator must develop this – seemingly – innate ability into a conscious competence. Above all, this requires knowledge of the culture-­contrastive dimension of modes, media, and genres. Multimodal translation studies, and thus an extension of the research interests to all modes and their transfers, does not mean relativising the linguistic dimension in its relevance for translation, but rather placing it in a larger context and thus ultimately delving deeper into the transcultural mechanisms of text composition and design. To this end, it would certainly be necessary to clarify its terminology with regard to modality and mediality, and to develop translation-­relevant analysis instruments for other modes. Linguistic theories as well as image-­and music-­theoretical approaches are suitable for this purpose. Finally, translator training should also be adapted to recognise multimodal competence as an integral part of translation competence. The required interdisciplinary competence builds on disciplinary competence, which manifests in a stringent understanding of the subject matter and a clearly defined research object. In this respect, translation studies are confronted with the question of how “translation proper” and, thus, its subject matter, is actually defined. Roman Jakobson’s answer – interlingual translation – is probably obsolete in the age of multimedia. In the twenty-­first century, the answer can only be that translation is a multimodal practice and translatorial action is a multimodal semiotic act.

Notes 1 Although Roman Jakobson (1959) introduced the category of “intersemiotic translation,” strictly speaking, his classification was not semiotic but linguistic. Later, explicitly semiotic definitions of translation (e.g., Gorlée 1994; Stecconi 2004) did not bring about a semiotic reorientation in mainstream translation studies, either. 2 For a more detailed overview, see Bird (2007). 3 Similarly, Hess-­Lüttich points to the “constructive gestalt” (1981, 324) of those signs that generate meaning only in the context of concrete communication processes. In this regard, he also sees the term “sign” as problematic. Decades later, Kress and van Leeuwen solved this problem by introducing the term “semiotic resource.” Cf. also van Leeuwen (2005, 95): “the term ‘resource’ is preferred, because it avoids the impression that ‘what a sign stands for’ is something pre-­g iven, and not affected by its use.” 4 For a more detailed examination of the communicative-­sociological contextualisation of signs, see Richter (1978).

66  Klaus Kaindl

5 “Mode” is an established term in various disciplines including linguistics, psychology, narratology, medicine, and, of particular relevance to us, in interpreting studies, where mode is used to describe the various forms of interpreting, e.g., simultaneous interpreting, consecutive interpreting, remote interpreting, etc., which can be problematic with regard to interdisciplinary work. 6 One of the first scholars to explicitly address multimodal texts in translation studies was Katharina Reiß (1971). 7 For an overview of the positioning of translation studies within various linguistic subdisciplines, see Kaindl (2004, 15–22). 8 For the various terms and the associated implications, see Kaindl (2013, 259–61). 9 The submodes of spoken language include volume, voice qualities, intonation, and speed; the submodes of written language are typography and layout; the submodes of images are colour, lines, and forms; and the submodes of music include rhythm, melody, harmony, and dynamics. Stöckl (2004, 14) also differentiates between “peripheral modes,” which result from the media realisation of the core modes. 10 For an overview of this research area and the different forms of intermediality, see Rajewski (2002). 11 Bateman (2008) examines genres from a multimodal perspective. 12 The distinction between mode and medium is not always clear, which is also evidenced by some inconsistencies in Kress and van Leeuwen’s classification. While, for example, colour is initially referred to as “semiotic mode” (Kress and van Leeuwen 2002), van Leeuwen later classifies it as “expression media” (2016, 109). 13 A detailed discussion of the translation with regard to genre, medium, and the effects on the relationship among language, music, and production can be found in Kaindl (1995, 188–206). 14 This definition is an expansion on Prunc’s (2004) definition, which is more language-­ centred, but very comprehensive in principle. 15 Newfield (2014) provides an overview of the nomenclature of intermodal transfer phenomena. 16 It is also debatable whether the term is a good choice. Although the term is etymologically related to translation, transduction is also a central concept in genomics research, where it describes the process by which foreign DNA is transported into a cell via a virus or bacterium. 17 For a detailed examination of Elvis Presley songs from 1956–69 with a particular focus on the verbal dimension, see Kaindl (2012); for the musical dimension of Hound Dog, see also Dorkin (2009, 94–100). 18 In the following I will focus on the verbo-­musical dimension. Of course, the visual mode, Presley’s body movements, his way of dancing, also had an impact on the reception of the lyrics and the music. 19 For a detailed overview of the social development context, see Szatmary (1996), and for musical characteristics, see Brown (1987). 20 For more details on the development and characteristics of Schlager music, see Port le roi (1998).

References Primary sources Presley, Elvis. 2002/2003. Elvis Forever. RCA/BMG. Presley, Elvis. n.d. Elvis singt deutsch Folge 3. Bear Records.

Secondary sources Baldry, Anthony, and Christopher Taylor. 2002. “Multimodal Corpus Authoring System: Multimodal Corpora, Subtitling and Phasal Analysis.” In Multimodal Resources and

A multimodal conception of translation  67

Multimodal Systems Evaluation, edited by Mark Maybury and Jean-­Claude Martin, 45–51. Las Palmas: LREC. Bar-­Hillel, Yehoshua. 1960. “The Present Status of Automatic Translation of Languages.” Advances in Computers 1: 91–163. Bassnett, Susan, and André Lefevere. 1990. “Introduction: Proust’s Grandmother and the Thousand and One Nights. A ‘Cultural Turn’ in Translation Studies.” In Translation, History and Culture, edited by Susan Bassnett and André Lefevere, i–xiii. London and New York: Pinter. Bateman, John. 2008. Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. New York: Palgrave MacMillan. Berressem, Hanjo. 2004. “Poststruktularismus.” In Grundbegriffe der Literaturtheorie, edited by Ansgar Nünning, 218–21. Stuttgart: Metzler. Bird, Alexander. 2007. “What Is Scientific Progress?” Nous 41 (1): 64–89. Borodo, Michal. 2015. “Multimodality, Translation and Comics.” Perspectives 21 (1): 22–41. Brown, Charles T. 1987. The Art of Rock and Roll, 2nd ed. Englewood Cliffs, NJ: Prentice Hall. Catford, John C. 1965. A Linguistic Theory of Translation. London: Oxford University Press. Semiotic Translation.” Cross, Judith Leah. 2016. “Enigma: Aspects of Multimodal Inter-­ Hermes 55: 91–103. Delabastita, Dirk. 1989. “Translation and Mass Communication: Film and TV Translation as Evidence of Cultural Dynamics.” Babel 35 (4): 193–218. De Pedro Ricoy, Raquel. 2012. “Multimodality in Translation: Steps Towards Socially Useful Research.” Multimodal Communication 1 (2): 181–203. Derrida, Jacques.1980. “La loi du genre.” Glyph 7: 176–201. Dorkin, Katharina. 2009. Die Übersetzung der musikalischen Dimension in der Popularmusik am Beispiel der Lieder von Elvis Presley. Wien: Unpubl. MA-­thesis. Eco, Umberto. 1972. Einführung in die Semiotik. München: Fink. Elleström, Lars. 2010a. “Introduction.” In Media Borders, Multimodality and Intermediality, edited by Lars Elleström, 1–8. New York: Palgrave MacMillan. ———. 2010b. “The Modalities of Media: A Model for Understanding Intermedial Relations.” In Media Borders, Multimodality and Intermediality, edited by Lars Elleström, 11–48. New York: Palgrave MacMillan. ———. 2017. “Transfer of Media Characteristics Among Dissimilar Media.” Palabra Clave 20 (3): 663–85. Gambier, Yves. 2006. “Multimodality and Audiovisual Translation.” In Audiovisual Translation Scenarios: Proceedings of the Second MuTraConference in Copenhagen 1–5 May, edited by Mary Carroll, Heidrun Gerzymisch-­Arbogast and Sandra Nauert. Accessed May 05, 2018. www.euroconferences.info/proceedings/2006_Proceedings/2006_Gambier_ Yves.pdf. Gentzler, Edwin. 2017. Translation and Rewriting in the Age of Post-­Translation. London and New York: Routledge. Gorlée, Dinda. 1994. Semiotics and the Problem of Translation:With Special Reference to the Semiotics of Charles S. Peirce. Amsterdam and Atlanta: Rodopi. Gottlieb, Henrik. 1994. “Diagonal Translation.” Perspectives 2 (1): 101–21. Hess-­Lüttich, Ernest W. B. 1978. “Semiotik der multimedialen Kommunikation. Eine Problemskizze.” In Angewandte Semiotik, edited by Tasso Borbé and Martin Krampe, 21–48. Wien: H. Egermann. ———. 1981. Grundlagen der Dialoglinguistik. Berlin: Erich Schmidt. Holz-­Mänttäri, Justa. 1984. Translatorisches Handeln: Theorie und Methode. Helsinki: Suomalainen Tiedeakatemia.

68  Klaus Kaindl

———. 1993. “Textdesign – verantwortlich und gehirngerecht.” In Traducere Navem: Festschrift für Katharina Reiß zum 70. Geburtstag, edited by Justa Holz-­Mänttäri and Christiane Nord, 301–20. Tampere: Tampereen Yliopisto. Hutcheon, Linda. 2006. A Theory of Adaptation. London and New York: Routledge. Hutcheon, Linda, and Siobhan O’Flynn. 2013. A Theory of Adaptation, 2nd ed. London and New York: Routledge. Iedema, Rick. 2003. “Multimodality, Resemiotization: Extending the Analysis of Discourse as Multi-­Semiotic Practice.” Visual Communication 2 (1): 29–57. Jäger, Ludwig. 2002. “Transkriptivität: Zur medialen Logik der kulturellen Semantik.” In Transkribieren: Medien/Lektüre, edited by Ludwig Jäger and Georg Stanitzek, 19–41. München: Fink. Jakobson, Roman. 1959. “On Linguistic Aspects of Translation.” In On Translation, edited by Reuben A. Brower, 232–39. Cambridge, MA: Harvard University Press. Jewitt, Carey. 2014a. “An Introduction to Multimodality.” In The Routledge Handbook of Multimodal Analysis, 2nd ed., edited by Carey Jewitt, 15–30. London and New York: Routledge. ———. 2014b. “Different Approaches to Multimodality.” In The Routledge Handbook of Multimodal Analysis, 2nd ed., edited by Carey Jewitt, 31–43. London and New York: Routledge. Jiménez Hurtado, Catalina, and Silvia Soler Gallego. 2013. “Multimodality, Translation, Accessibility: A Corpus-­Based Study of Audio Description.” Perspectives 21 (4): 577–94. Kade, Otto. 1968. Zufall und Gesetzmäßigkeit in der Übersetzung: Beiheft zur Zeitschrift Fremdsprachen I. Leipzig:VEB Enzyklopädie. Kaindl, Klaus. 1995. Die Oper als Textgestalt: Perspektiven einer interdisziplinären Übersetzungswissenschaft. Tübingen: Stauffenburg. ———. 2004. Übersetzungswissenschaft im interdisziplinären Dialog: Am Beispiel der Comicübersetzung. Tübingen: Stauffenburg. ———. 2012.“ ‘Little Linda hör zu’: Rock ‘n’ Roll Gender und Übersetzung.” In Translationswissenschaftliches Kolloquium II: Beiträge zur Übersetzungs-­und Dolmetschwissenschaft (Köln/ Germersheim), edited by Barbara Ahrens, Silvia Hansen-­Schirra, Monika Krein-­Kühle and Michael Schreiber, 307–25. Frankfurt am Main: Peter Lang. ———. 2013. “Multimodality and Translation.” In Routledge Handbook of Translation Studies, edited by Carmen Millán and Francesca Bartrina, 257–70. London and New York: Routledge. Ketola, Anne. 2016. “Towards a Multimodally Oriented Theory of Translation: A Cognitive Framework for the Translation of Illustrated Technical Texts.” Translation Studies 9 (1): 67–81. Kloepfer, Rolf. 1976. “Komplementarität von Sprache und Bild.” Sprache im technischen Zeitalter 57: 42–56. Kress, Gunther. 2000. “Design and Transformation. New Theories of Meaning.” In Multiliteracies: Literacy Learning and the Design of Social Futures, edited by Bill Cope and Mary Kalantzis, 153–61. Melbourne: Palgrave MacMillan. ———. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. London and New York: Routledge. Kress, Gunther, and Carey Jewitt. 2003. “Introduction.” In Multimodal Literacy, edited by Gunther Kress and Carey Jewitt, 1–18. New York: Peter Lang. Kress, Gunther, and Theo van Leeuwen. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Hodder Arnold. ———. 2002. “Colour as a Semiotic Mode – Notes for a Grammar of Colour.” Visual Communication 1 (3): 343–69.

A multimodal conception of translation  69

Kristeva, Julia. 1968/1977. “Semiologie – kritische Wissenschaft und/oder Wissenschaftskritik.” In Textsemiotik als Ideologiekritik, edited by Peter V. Zima, 35–51. Frankfurt am Main: Suhrkamp. Krystallidou, Demi. 2016. “Investigating the Interpreter’s Role(s): The A.R.T. Framework.” Interpreting 18 (2): 172–97. Kuhn, Thomas S. 1970. The Structure of Scientific Revolutions. Chicago: Chicago University Press. Lee,Tong-­King. 2012. “Performing Multimodality: Literary Translation, Intersemioticity and Technology.” Perspectives 21 (2): 241–56. Lefevere, André. 1992. Translation, Rewriting and the Manipulation of Literary Fame. London and New York: Routledge. Mavers, Diane. 2011. Children’s Drawing and Writing: The Remarkable and the Unremarkable. London and New York: Routledge. McLuhan, Marshall. 1950. “Foreword.” In Empire and Communications, edited by Harold A. Innis, v–xii. Toronto: University of Toronto Press. ———. 1964. Understanding Media:The Extensions of Man. New York: McGraw-­Hill. Middleton, Richard. 1977/1992. “All Shook Up? Innovation and Continuity in Elvis Presley’s Vocal Style.” In The Elvis Reader: Texts and Sources of the King of Rock ‘n’ Roll, edited by Kevin Quain, 3–12. New York: St. Martin’s Press. Munday, Jeremy. 2014. “Advertising: Some Challenges to Translation Theory.” The Translator 10 (2): 199–219. Newfield, Denise R. 2014. “Transformation, Transduction and the Transmodal Moment.” In The Routledge Handbook of Multimodal Analysis, 2nd ed., edited by Carey Jewitt, 100–13. London and New York: Routledge. Nida, Eugene A., and Charles R. Taber. 1969. The Theory and Practice of Translation. Leiden: E. J. Brill. Oittinen, Riitta, Anne Ketola, and Melissa Garavini. 2017. Translating Picturebooks: Revoicing the Verbal, the Visual and the Aural for a Child Audience. London and New York: Routledge. Pérez-­González, Luis. 2014. “Multimodality in Translation and Interpreting Studies: Theoretical and Methodological Perspectives.” In A Companion to Translation Studies, edited by Sandra Bermann and Catherine Porter, 119–31. Chichester: Wiley-­Blackwell. Popper, Karl R. 1998. Objektive Erkenntnis: Ein evolutionärer Entwurf, 4.Verbesserte und ergänzte Auflage. Hamburg: Hoffmann und Campe. Port le roi, André. 1998. Schlager lügen nicht: deutscher Schlager und Politik in ihrer Zeit. Essen: Klartext. Posner, Roland. 1976a. “Semiotische Paradoxien in der Sprachverwendung – Am Beispiel von Sternes ‘Tristam Shandy’.” Sprache im technischen Zeitalter 57: 25–41. ———. 1976b. “Zum Thema.” Sprache im technischen Zeitalter 57: 2–4. Poulsen, Søren Vigild. 2017. “The ‘Same’ Meaning Across Modes? Some Reflections on Transduction as Translation.” In New Studies in Multimodality. Conceptual and Methodological Elaborations, edited by Ognyan Seizov and Janina Wildfeuer, 37–64. London: Bloomsbury. Prunc, Erich. 2004. “Zum Objektbereich der Translationswissenschaft.” In Und sie bewegt sich doch…: Translationswissenschaft in Ost und West. Festschrift für Heidemarie Salevsky zum 60. Geburtstag, edited by Ina Müller, 263–85. Frankfurt am Main: Lang. Rajewski, Irina O. 2002. Intermedialität. Tübingen: A. Francke. Reiss, Katharina. 1971. Möglichkeiten und grenzen der übersetzungskritik: Kategorien und kriterien für eine sachgerechte beurteilung von übersetzungen. München: Hueber.

70  Klaus Kaindl

Richter, Helmut. 1978. “Zum kommunikationssoziologischen Inhalt des Medienbegriffs.” In Stilforschung und Rhetorik, Patholinguistik und Sprecherziehung, edited by Ernest W. B. Hess-­Lüttich, Günter Peuser, and Helmut Geißner, 37–43. Stuttgart: Hochschulverlag. Ryan, Marie-­Laure, and Jean Noël Thon. 2014. “Storyworlds Across Media: Introduction.” In Storyworlds Across Media: Toward a Media-­Conscious Narratology, edited by Marie-­Laure Ryan and Jean Noël Thon, 1–21. Lincoln and London: University of Nebraska Press. Stecconi, Ubaldo. 2004. “Interpretive Semiotics and Translation Theory: The Semiotic Conditions to Translation.” Semiotica 150 (1–4): 471–89. Stöckl, Hartmut. 2004. “In Between Modes: Language and Image in Printed Media.” In Perspectives on Multimodality, edited by Eija Ventola, Cassily Charles and Martin Kaltenbacher, 9–30. Amsterdam: Benjamins. Szatmary, David P. 1996. A Time to Rock:A Social History of Rock and Roll. New York: Schirmer Books. Toury, Gideon. 1994. “A Cultural-­Semiotic Perspective.” In Encyclopaedic Dictionary of Semiotics, Vol. 2 N-­Z, edited by Thomas A. Sebeok, 1111–24. Berlin and New York: Mouton de Gruyter. ———. 1995. Descriptive Translation Studies – and Beyond. Amsterdam and Philadelphia: Benjamins. van Doorslaer, Luc, and Laurence Raw. 2016. “Adaptation Studies and Translation Studies: Very Interactive Yet Distinct.” In Border Crossings: Translation Studies and Other Disciplines, edited by Yves Gambier and Luc van Doorslaer, 189–204. Amsterdam and Philadelphia: John Benjamins. van Leeuwen, Theo. 2005. Introducing Social Semiotics. London and New York: Routledge. ———. 2016. “A Social Semiotic Theory of Synesthesia? A Discussion Paper.” Hermes 55: 105–19. Vermeer, Hans J. 1986. “Übersetzen als Kultureller Transfer.” In Übersetzungswissenschaft – eine Neuorientierung: Zur Integrierung von Theorie und Praxis, edited by Mary Snell-­Hornby, 30–53. Tübingen: Francke. Walther, Elisabeth. 1997. “The Sign as Medium, the Medium Relation as the Foundation of the Sign.” In Semiotics of the Media: State of the Art, Projects and Perspectives, edited by Winfried Nöth, 79–86. Berlin and New York: Mouton de Gruyter. Wyatt-­Smith, Claire, and Kay Kimber. 2009. “Working Multimodally: Challenges for Assessment.” English Teaching Practice and Critique 8 (3): 70–90. Yuste Frías, José. 2012. “Fun for All: Videojuegos y paratraducción.Video Game’s Translation and Paratranslation.” In Blog de Yuste: On y sème à tout vent. Vigo: Blog de Investigación T&P, 23/03/2012. Accessed May 5, 2018. www.joseyustefrias.com/index.php/blog/ item/fun-­for-­all-­1.html.

3 MEANING-­(RE)MAKING IN A WORLD OF UNTRANSLATED SIGNS Towards a research agenda on multimodality, culture, and translation Elisabetta Adami and Sara Ramos Pinto

1. Introduction Multimodality has always been “the normal state of human communication” (Kress 2010, 1). From face-­to-­face interaction combining resources such as gesture, body movement, and face expression, to disembodied representations combining image with writing, multimodal texts (and the need to translate them) have always been integral to (intercultural) communication. In recent years, technological development has made this only the more manifest by facilitating multimodal text production in a myriad of platforms. This has led disciplines like translation studies (TS), traditionally focused on the verbal, to turn to the recent field of multimodal studies (MS) for adequate methodologies and analytical frameworks that can ground research on the translation of multimodal texts such as films, websites, or comics (for instance, see the chapter by Luis Pérez-­González in this volume, Chapter 4). A new multimodal turn in TS has adopted some of the analytical tools and concepts developed within the area of multimodality, yet with little in-­depth reflection on the principles upon which those tools have been built or on what accepting those principles means for TS. As a result, translation theories remain mostly focused on the verbal while limiting the discussion of all other semiotic resources (integral to meaning-­making in multimodal texts) to a contextualising role. The link between TS and MS is – or should be – bidirectional. Globalisation and the consequent rising transnational circulation of goods, people, and cultural artefacts has increasingly exposed us to visual, auditory, and material resources produced in other areas of the world, while contributing to a multiplication and fragmentation of communities with specific sign-­making practices (New London Group 1996). This questions the idea of homogenous nation-­and language-­bound communities sharing the same cultural background for meaning interpretation.Yet MS has not yet tackled

72  Elisabetta Adami and Sara Ramos Pinto

the issue of “culture” in relation to resources other than language. Not only is there a lack of studies on the use of semiotic resources with a cross-­cultural comparative take as well as in intercultural contexts, but also, while it is assumed that modes are culture-­ specific resources (Kress 2010), the notion of culture remains under-­defined and under-­explored (Adami 2017; Hawkins 2018). MS has yet to focus on the meaning-­ making dynamics of transnational circulation, that is, on how multimodal resources are (re)interpreted and (re)signified when they move across place, time, and social groups, including “small and big cultures” (Holliday 1999). changing social semiotic landscape, both Because of the changed and fast-­ MS and TS need to start working on new assumptions and address a series of disciplinary gaps, which can be filled only through a joint transdisciplinary enterprise. In this chapter we aim to sketch a joint semiotic and translation research agenda, to develop understanding of meaning-­making practices in a changed, increasingly transnational, social semiotic landscape. We conceive this undertaking as a pre-­ requisite to derive implications for TS and provide indications that can support translation practice and training in today’s communicative landscape. We will start by revising some of MS’s principles in order to identify the gaps in and most immediate issues for social semiotic research.We will then discuss the implications for TS of adopting a multimodal approach. In this effort, we will review some of the basic concepts of TS and raise several questions we believe should be at the center of the discipline. We will finish by proposing the sketch lines of a research agenda that would allow MS and TS to address the issues previously brought forward.

2. Issues for a way forward in social semiotic research As discussed in the Introduction, by moving away from structuralist semiotics centred on codes, social semiotics (Hodge and Kress 1988; van Leeuwen 2005; Kress 2010) elaborated the notion of multimodality and introduced the concept of “mode,” following Halliday’s (1978) use of the term to distinguish between speech and writing. Modes are conceived as “a socially and culturally shaped set of resources for making meaning” (Bezemer and Kress 2008, 6), which have different affordances deriving from their materiality and the ways they have been shaped historically in different social groups to fulfil specific communicative needs. Social semiotic research has undermined the assumption of the arbitrariness of language (Kress 1993) vs. an assumed iconicity of nonverbal resources. Relevant to our discussion is the understanding that resources like image, often referred to as iconic, do not have a more “natural” relation (of resemblance) with the world than language and do not make meaning universally. The meaning potentials of any given semiotic resource result from the history of its past uses in given social groups; hence, knowledge is required for its interpretation, or better, to formulate hypotheses on the meaning expressed by a given use of the resource (given that, in each specific instance, sign-­makers have agency in associating a given form as most apt to represent the meaning that they want to express). As two banal examples, the floppy-­disk-­shaped button in software interfaces has no more “natural” or

Meaning-­(re)making  73

FIGURE 3.1 

Bleeding nose used in anime to signify sexual arousal

Source: (Original anime: Karin)

“universal” resemblance to its signified than its equivalent written label “save.” To understand the meaning of an anime character’s sudden bleeding nose (Figure 3.1), viewers need to know the “conventions” for that specific use of image in anime, used to signify “sexual arousal.” These conventions are neither arbitrary nor the result of natural resemblance; in both cases, the relation between the signifier/form and its signified/meaning is motivated (Kress and van Leeuwen 1996; Kress 2010), and the motivation lies as much in the materiality as in the past uses of that specific signifier in given social groups.1 Dismantling a presumed universally understood relation of resemblance with the world for resources other than language brings two issues to the fore, namely the need for a redefinition of “context” vs. “co-­text” and that of shared/non-­shared “culture,” as semiotically conceived. Let us examine each in detail.

2.1  The “context” vs. “co-­text” approach The first issue concerns the question of conceiving of resources that come together with speech and writing as either “context” or “co-­text.” In this work, we use “text” to define any multimodally composed meaningful whole (or multimodal ensemble), rather than restricting it to writing. By “context” we mean the social semiotic environment for the design, production, distribution/circulation of, and engagement with so-­defined texts (including participants in each of these processes, and the social and semiotic resources available to make signs and meanings); by “co-­ text” we mean signs (in any mode and their combinations) co-­occurring with those that are the momentary focus of attention in a text. Linguistics and translation traditionally adopt the “context” approach and consider what they normally term as “nonverbal”2 resources not as signs (forms

74  Elisabetta Adami and Sara Ramos Pinto

associated with meaning) but as a mere issue of cultural reference. In this approach what is shown visually, for example, is (rather than represents) a certain phenomenon in the world, which may be more or less culture specific or shared. In turn, social semiotics treats all communicative manifestations as signs. This view adopts a “co-­text approach,” which considers all resources co-­occurring with writing or speech as signifying elements that make meaning on their own and in relation to each other. Rather than cultural references, semiotic resources are conceived as forms of expression (i.e., signifiers that are associated to meanings/ signifieds to make signs), fully capable of a range of meaning potentials, which have accumulated through past uses among specific social groups. These socially shaped (culture-­specific) meaning potentials are not only denotational but also affective, identity shaping, and constructing of register, tone and style, mood and modality, as well as cohesion and coherence within the overall representation. Taking the example of the anime bleeding nose (Figure 3.1), the way the bleeding is represented (as a burst or as a dropping, etc.) and its relation with the character’s facial expression and body movement modulate the (sexual arousal) feeling expressed as well as the character’s positioning towards it; its relation with the soundtrack and other filmic effects (e.g., stills, flashes etc.) shapes the overall tone of the scene (e.g., as dramatic, romantic, or humorous). Adopting either the context or the co-­text approach involves crucial epistemological differences and consequences in terms of the decisions to be made in translation. Following the “context” approach, when considering, for example, visuals to be circulated to different audiences, it is a matter of deciding whether and how to fill the gap of shared/background knowledge needed to understand the cultural reference, as when facing cultural references in language; it is in sum a matter of knowledge about the world that is depicted in the visual (i.e., cultural knowledge), rather than knowledge of how the visual shapes the world that it depicts (i.e., semiotic knowledge). When adopting the “co-­text” approach, by considering all semiotic resources as signs, questions open not only for the cultural knowledge needed to understand the referential/denotational meaning (i.e., to grasp the cultural reference). They involve the required semiotic knowledge, i.e., the knowledge needed to understand how resources are used to make meaning at all levels. This means considering the meaning potentials of specific gestures, facial expressions, proxemics, music, filmic effects, camera angles, colour, clothing, and so on (as well as their combination) to represent reality, shape affect, identity, politeness/distance, formality/informality, and to modulate truth values, for example. In sum, in considering meaning as multimodally constituted and orchestrated, the co-­text approach assumes complementarity of resources in a multimodal complex; hence, in a representation, all resources are used following or innovating from past uses for meaning-­making in all its aspects. These interact with all others present, requiring translators to make holistic multimodal choices as to what needs to be translated, from which modal resource, into which other modal resource, on the basis of an assessment of the target audience’s semiotic knowledge in all

Meaning-­(re)making  75

modes, rather than solely of their knowledge of the cultural references of the source text. From this derives a second issue, which is yet to be tackled in social semiotic research, as to the mapping of semiotic knowledge across “cultures,” and hence to defining culture in semiotic terms.

2.2  Semiotic knowledge and the issue of culture If, to understand each other, meaning-­makers need to share (some extent of) semiotic knowledge, that is, knowledge of semiotic practices in all modes at play (rather than only cultural references), then the issue of what is shared/non-­ shared in any form of expression achieves primary significance. This is true in all communicative events (as when assessing how an interlocutor will interpret a certain gesture in face-­to-­f ace interaction), and becomes crucial when designing or adapting multimodal texts for transnational circulation, since we can hypothesise that as circulation widens, so the range of meaning possibilities increases (Rymes 2014). A main assumption to be questioned is that boundaries in shared uses of semiotic resources such as image or music match linguistic boundaries/communities; this assumption is not grounded empirically, yet it is widely present both in studies on intercultural communication and in localisation practices (e.g., when replacing images on different language versions of a company website). Questioning (or verifying) it involves considering the historically different dynamics of circulation of resources other than (what has been normalised as) language. This has been underexplored in social semiotic research, while disciplines such as musicology or iconography have specialised only on individual modes; yet it is, we argue, of primary importance for the examination of the relation between translation and multimodality in today’s globalised world.

2.2.1 Language as an exception: National codification and translation tradition Nation-­states have historically devoted extensive political, economic, and educational effort into codifying language use among their citizens,3 through a series of homogenising forces, including dictionaries, grammars, and literacy enterprises through the education system and national mass media. Ideologically shaped notions of “national languages” (or standards) have been constructed through selective normativity on the mode of writing (imposed also onto the mode of speech, particularly in prestige forms and formal contexts, as “correct” usage). National codification and standardisation of linguistic form/meaning associations have been strengthened also through cross-­linguistic comparison, the production of bilingual dictionaries, and foreign language education. By establishing systems of equivalence between national languages, these mono-­, bi-­, and multilingual literacy enterprises have taught us how to read, interpret, and use linguistic resources, thus constraining their range of meaning-­making possibilities.4

76  Elisabetta Adami and Sara Ramos Pinto

Other semiotic resources have not been subject to the same systematic extent of national codification. Nation-­states have not put the same effort into codifying gestures, music, or filmic conventions as in codifying writing. The propagation of hegemonic uses of different semiotic resources varies, so certain modes have generated conventions of use in certain practices because of their professionally oriented centres of prestige (e.g., influential groups of film-­makers, photographers, musicians, graphic or fashion designers), while certain others, like gestures and facial expressions, have developed more through face-­to-­face contact (and then circulated in visual representations), possibly standardised as best/correct practices of behaviour in certain contexts (e.g., “etiquette”).Yet, centres of prestige for sign-­making practices in nonverbal modes have historically been more transnationally connected. Descriptions of “prestige/authoritative” uses of specific semiotic resources from authoritative practitioners and critics (e.g., manuals and treatises on graphic design, photography, film, architecture, and music) have normally originated and circulated from internationally influential centres/schools/traditions (consider, for “Western” areas of the world, e.g., the reach of Christian iconography for image since pre-­ nation-­state times, the circulation of dress codes around Europe and through colonisation, up to, e.g., Hollywood for films). More significantly, nonverbal modes did not undergo national standardisation through literacy enterprises, and have been therefore less influenced by national homogenising forces and systems of equivalence in form-­meaning associations. Only writing (and speech), particularly in its ideological notion of “national language,” has systematically been subject to a history of (intra-­semiotic) translation. Translation of language too, while mediating between different languages, has paradoxically also contributed to strengthening the divide between national linguistic communities, who received representations mediated into “their” language, rather than being exposed to the language of “others” and forced to make (their own) meaning out of it, as happened with visual representations or music, for example. The exceptional history of codification and translation that language has undergone has ultimately contributed to determining different dynamics of circulation and meaning-­making of language than of other resources.

2.3  Meaning-­making in a world of untranslated signs If we adopt a semiotic perspective and conceive of anything in our environments as a potential sign, we come to realise that we live in a communicative world of mainly untranslated signs; not only is language an exception, because of its historical national codification processes, but so is translation, as a form of cultural and semiotic mediation. With the exception of writing and speech, we normally make meaning without anybody or anything mediating between us and the semiotic environment. Rather than making signs in nonverbal resources more universally understood, this makes interpretation of form/meaning association at the same time more individualised and shared within social groupings that cross national boundaries.

Meaning-­(re)making  77

Hence the coincidence of semiotic knowledge in these modes with a nationally defined linguistic knowledge is a problematic assumption (and ultimately reinforces an essentialist view of culture, long-­time dismissed in culture research). Further enhanced by the transnational and fragmented/multiplied character of communities and “affinity spaces” (Gee 2005) in today’s globalised world, a given target audience identified for language translation cannot be assumed to be homogeneous in its interpretation of other resources. With blurred boundaries between foreign and local signs, intersectional variables may be more significant in driving interpretation, along with individuals’ personal trajectories, in terms of which communities of practice/interest and affinity spaces specific individuals participate in, and of which sign-­making practices they have been more exposed to and have become more familiar with, and hence have entered their semiotic (rather than linguistic-­ only) repertoires. In the example of the anime nosebleed mentioned earlier, its association with “sexual arousal” may be known to anime fans speaking different languages and yet not known to a Japanese speaker who comes across that sign in anime for the first time. In other terms, sharing (more or less) the “same” language does not necessarily coincide with the sharing of semiotic knowledge for image or music, for example, and the meaning the latter two construct in their relation with language. Adopting a co-­text approach to meaning as multimodally constituted and considering issues of shared/non-­shared semiotic knowledge across linguistic communities (because of the different circulation dynamics of resources other than language) has far-­reaching implications both for MS and TS. On the one hand, while what we label here a co-­text approach is a long-­established assumption in MS, the issues of transnational circulation and national codification have not been adequately reflected upon in the field, precisely because of its little concern so far with translation as broadly defined; in this sense, a joint discussion and investigation with TS (eminently concerned with issues of “culture”) can help social semiotic research to fill the gap, as proposed in the research agenda in section 4. On the other hand, adopting a co-­text approach and considering shared/non-­shared semiotic knowledge in all resources constitutes a crucial epistemological shift for TS, with far-­reaching implications not just for research, but also for translation practice and training. In section 3 we discuss how adopting such an approach forces us to broaden the limits of translation and revisit some of the most fundamental concepts in TS.

3. Issues for a way forward in translation To acknowledge the multimodal nature of communication means accepting that the realms of translation and TS include more than words in context. It means accepting that translation needs to consider all modes and the meanings they promote (on their own and in relation to other modes) but also that all resources co-­occurring with writing/speech are signs in their own right that might present different challenges to (different) viewers. This constitutes a fundamental shift which implies a

78  Elisabetta Adami and Sara Ramos Pinto

different understanding of translation and leads us to revisit fundamental concepts such as “text,” “source text” and “target text,” or “equivalence.” It also opens the question on whether, as socioculturally shaped resources, modes other than writing and speech might need to be translated and how that could be achieved. The discussion on the (non-­) translation of nonverbal resources (which is addressed, in different ways, by Marcus Tomalin (Chapter 6), Helen Julia Minors (Chapter 7), and Tamarin Norwood (Chapter 8) in this volume) comes hand-­in-­hand with issues to do with the aesthetics, authorship, prestige, and function of multimodal texts, which are in fact some of the issues we would like to raise in this chapter. Some of these questions have been asked by the first scholars in the field of Audiovisual Translation (AVT), who tried to reflect on what it meant for TS to expand its basic concepts and include products composed by speech/writing along with image, sound, or movement (Gambier 2003). The discussion, however, has not matured much beyond the initial questioning, and nonverbal resources have remained reductively considered only as contextual elements (see section 2.1). Multidisciplinary approaches have been suggested (Cattrysse 1992; Remael 2001; Gambier 2013; Pérez-­González 2014), and MS is one of the areas that researchers have turned to on their quest for new analytical tools/frameworks purposely developed to examine multimodal texts. However, taking advantage of the analytical frameworks of a different area of study demands reflection on the implications deriving from the assumptions underpinning those frameworks. In the next section we will reflect on the implications that the issues introduced in Section 2 have for TS.

3.1  Multimodality and the concept of translation As a first crucial question, if meaning is achieved through intermodal relations that are culturally specific, how can we assume that the target audience can access such meaning when only one of those modes is translated? Setting any reservations around issues of “meaning transfer” (for which we may prefer Kress’s phrase “transposing meaning” – see his chapter in this volume, Chapter 1) and “equivalence” aside, it must be recognised, however, that practices translating only the verbal (e.g., subtitling, dubbing, translation of comics) implicitly or explicitly assume that (i) nonverbal resources are universal and easily interpreted without any further mediation; and that, as a result, (ii) no new intermodal relations and meanings are introduced when the verbal resources are replaced by verbal resources in another language. These assumptions seem difficult to reconcile with the founding notion that translation is about meaning in context, leading us to revisit concepts such as “text,” now including multimodal products, as well as “source text” and “equivalence,” given that all modes are included in what is considered for translation, and equivalence is sought for verbal and nonverbal modes alike as well as the meanings erected through intermodal relations. Such logocentric assumptions do not seem to be true in all areas of translation or to all translation practitioners. Those involved in fansubbing, for example, openly recognise the role played by nonverbal resources in meaning creation along with

Meaning-­(re)making  79

the difficulty users might have in interpreting them. The subtitling developed in this context has challenged professional practice (deemed unable to account for the complexity and multitude of levels of meaning put forward by the source text) and boasts an innovative set of mediating/translation procedures of verbal and nonverbal resources alike. Another case in point, within professional practice, is localisation. In this context, a high-­level manipulation of the source text is accepted and strategies of mediation/translation are extended to nonverbal resources (alongside verbal ones), including, at times, the replacement of source nonverbal resources by target nonverbal resources, if deemed necessary in the context of the translation brief. Even if only at a glance, the discussion of these two cases immediately reveals a symptomatic difference between professional and non-­professional practice with regard to their understanding of translation. Non-­professional translation results from a widening of the concept of translation beyond the verbal. Professional practice either does not consider the potential need to translate meaning nonverbally expressed or, when it does, it does not consider it within the boundaries of translation. Revealing in this sense is the fact that localisation is acknowledged not as a form of translation (Esslink 2000) but as a broader process that includes the translation of the verbal mode and the “adaptation” of nonverbal modes. While replacing one language with another is considered as a necessary change, mediating/ translating an image (whatever the procedure used), for example, is considered to be a high-­level intervention, which is referred to in the industry with words such as “creative,” “free,” and “adaptation” (Jiménez-­Crespo 2013). The development of different terms certainly indexes the need to distinguish a type of translation in which function and commercial purpose are the priorities, but it also expresses the need for a different set of knowledge and skills than the ones translators traditionally have, as well as a different set of tools and the extra associated costs. In this context, translation remains defined within an understanding of equivalence limited to the verbal mode and one in which other resources, given their assumed iconicity/ universality, are to be engaged with directly, in an unmediated way, or else their “original” meaning would be lost. However, the consideration of resources other than the verbal, the different possible levels of intervention according to translation brief, prestige, or symbolic capital associated to the author or source text are something that translation theory should be able to account for within functional approaches to translation. After all, the existence of different translations following different strategies to cater to different audiences or fulfil different functions is not a new phenomenon for translation studies. Yet, to extend such considerations to resources other than the verbal, a new paradigm or conceptualisation of translation is needed. Further research on meaning-­making practices is needed to allow the discussion on translation and equivalence to broaden its scope and consider not only the meanings expressed by all resources but also the possible unexpected results of current translation practices. Target multimodal texts might be assuming a different profile than the one expected due to the fact that, contrary to common assumption, the newly translated verbal resources establish a new multimodal ensemble

80  Elisabetta Adami and Sara Ramos Pinto

in the target context when co-­occurring with non-­mediated nonverbal resources. Another important aspect calling for further conceptual and empirical analysis is the fact that recent technological developments have made manipulation of nonverbal resources both easier than before and a common practice with a growing number of text types. Replacing source images by target images is recurrently being used in the localisation of animated films (as in the case of Doraemon, in which a bento box was replaced by a pizza, Chaume 2016), without the corresponding theoretical and empirical analysis to ground the mediation process.

3.2  Translation of meaning rather than elements Another crucial question deriving from considering meaning as multimodally constituted is then “what” to translate when a text needs to reach a different audience. Considering elements such as images, gestures, or sounds as socially situated signs means acknowledging that (i) their meaning is positioned in time and space; (ii) it is mediated by specific traditions of use within that specific (target) context (which in turn shape audience’s expectations/interpretations); and (iii) part of what is being expressed comes from the purposeful selection and organisation of specific resources in the context of a given product. The reality instead is that target audiences are left to make meaning without any type of mediation between them and what can be very complex semiotic environments. Different levels of meaning pose specific challenges to viewers from a different context. Drawing on previous work (Kovačič 1995; Chesterman 2005, 2007; Gambier 2013; Ramos Pinto 2018), we propose three levels of meaning, adapted from the “representational,” “interpersonal,” and “textual” metafunctions used in social semiotics. The first level is the most obvious one, i.e., what is being represented, and the difficulties the audience might have in identifying a given gesture or object, like a floppy disk in a 1980s advert, for example. The inability of a younger generation to identify floppy disks has even been used for humorous purposes, as seen in a popular Internet meme with the sentence “I showed my 12 year old son an old floppy disk . . . He said: ‘WOW . . . Cool . . .You 3D printed the save icon!’ ”5 Visual and aural elements, for example, might often allow for a more immediate first-­level identification by appealing to the audience’s knowledge of the world; however, this may not always be true, either because the sign is not part of the audience’s context (like floppy disks for 12-­year-­olds), or because there is no natural resemblance with the world (e.g., blue curved stripes used to signify [waves of] freshness in a toothpaste package). The second level refers to the social meaning associated to a given object, sound, or gesture and the difficulties an audience might face not because they cannot identify the referents in question but because they might not share the necessary socially shaped semiotic knowledge to interpret them. In the television series The Big Bang Theory (s11, ep22), the character Amy, after trying on several wedding dresses that, following (Western) mainstream socio-­cultural fashion trends, could be interpreted as “modern/fashionable/sexy” (Figure 3.2), decides on a dress (Figure 3.3) that

Meaning-­(re)making  81

FIGURES 3.2, 3.3, and 3.4 

Amy, Penny, and Bernadette go wedding dress shopping

Source: (The Big Bang Theory, s11, ep22, 00:07:39–00:08:22)

causes surprise to her friends (Figure 3.4). The viewer is only able to understand this reaction if he/she is able to interpret the chosen dress’s social meaning as “old-­fashioned/tacky/not-­sexy.” Notwithstanding the different affordances of subtitling and dubbing, the challenge for translation thus comes from the fact that, even when nonverbal resources build on a shared resemblance with the world (identifying the dress as a wedding dress would not be challenging to most viewers), presenting them as in the source text without further mediation might not ensure cross-­cultural transfer at the level of social meaning: this type of wedding dress can be instead common in certain social groups, who, if not accustomed to current Western fashion trends, may interpret it as “beautiful” and find it challenging to understand Amy’s friends’ reaction. Perhaps more importantly, due to the common belief in the universality of nonverbal resources and the common practice of being left alone to interpret them, the audience may not recognise the existent “cultural bump” (Leppihalme 1997) and proceed interpreting on the basis of the social meaning promoted in the target context. This is the case not only for laypersons but also for translators whose training does not traditionally include multimodal analysis as discussed in this chapter. Translation practices such as professional subtitling seem to deny such potential for confusion, but practices such as localisation and fansubbing together with recent reception data (Chiaro 2014) seem to point otherwise. The final level refers to the meaning resulting from the intermodal relations established between the different modes for specific diegetic purposes. In the wedding dress scene, the verbal mode with Amy saying “This is it. This is the one!”, combined with the dress-­as-­sign, with its specific social meaning, and the friends’ facial expressions of surprise, produce a comedic moment (supported by a live audience’s laughter) that reinforces specific diegetic meaning, namely Amy’s characterisation as “geek” and hence peculiarly not associated with the social world of the other characters (which viewers are required to recognise).6 The challenge thus comes from the need to ensure that the audience is able to identify not only the elements in the image but also their social and diegetic meanings. This discussion has not had the attention it deserves in professional practice; even translation practices focused on accessibility such as Subtitling for the Deaf and Hard of Hearing and audio description (which have long called attention

82  Elisabetta Adami and Sara Ramos Pinto

to the importance of nonverbal modes) often show a strong focus on describing sounds or images to ensure identification, undervaluing considerations of the audience’s (in)ability to interpret the social or diegetic meaning even after identification has been made possible. The intricate relationship between the different modes brings an additional challenge related to the difficulty in distinguishing the contribution provided by each mode and to have clearly defined translation units. Notwithstanding the doubts raised by Stöckl (2004) regarding the independent existence of contributions, one possible way forward might come from results in perceptual psychology, which suggest that perception is selective, i.e., that “we attend to objects that bear salient meaning for certain goals” (Gibson 1979, 48). Furthering our understanding of relevance and narrative salience, expanding it to meaning produced multimodally (cf. Section 4) could be helpful in this context, as well as integrating the contribution from specific disciplines for different genres (like film studies and media studies).

3.3  Translation without nationally codified resources Alongside the “what” to translate comes the issue of “how” to translate nonverbal resources. In localisation practices, source nonverbal elements are often replaced by other elements of the same mode more familiar to the target audience (e.g., images or colours used in packaging). Similar procedures of substitution involving image manipulation are sometimes used in other genres such as comics (e.g., when someone bangs Asterix in the head, he sees “birds” in the French edition and “stars” in the Portuguese, Zanettin 2014) or children’s literature (e.g., Arabic editions of Disney’s Cinderella often present female characters wearing a hijab, Zitawi 2014). Non-­professional practices such as fansubbing present other solutions by taking advantage of headtitles and pop-­up balloons for annotation of nonverbal meaning or different colours for character or tone identification (Pérez-­González 2014). While localisation seems to assume that nonverbal resources are to be translated by target resources of the same nature, the use of colours and verbal headtitles in fansubbing suggests that the meaning expressed in one mode can be translated by resources in a different mode. These practices, however, have been (and still are) developed on the basis of intuition and most often not by translators or trained professionals. To enable informed decisions, data are needed on the effects of existing and new translation strategies onto meaning-­making. With the lack of national codification of nonverbal resources, translators cannot draw on a system of equivalence to inform translation practice or a translation tradition against which positioning their choices. Some genres have developed specific conventions of use that audiences assimilate through exposure; besides the already mentioned example of anime, comics developed a tradition on reading direction and representation of sounds (Zanettin 2014), and the same applies to expectations built up by music in horror films (Bordwell and Thompson 1979). Such convention development does not follow the language boundaries in which the translation and distribution industry is organised. The multimodal nature of texts seems thus to

Meaning-­(re)making  83

bring additional layers of complexity to the already challenging concept of “target audience”; the heterogeneous character of a product’s audience is multiplied, given that this might differ in semiotic knowledge, making it more difficult to make decisions, and leaving the translator at the crossroads of different traditions. The lack of research on shared/non-­shared semiotic knowledge will continue to leave practitioners without a way forward or enough data to consider the implications of their decisions.

3.4  Translation of nonverbal resources as a mediated activity As with any decision-­making process in translation, the (non-­)translation of nonverbal resources will also be mediated by contextual factors. As previously mentioned, the technological development of recent years has opened a number of options ranging from image manipulation to pop-­up balloons and dynamic subtitles, meaning that the nonverbal nature of certain resources no longer immediately constrains the mediation processes or define the translation strategy (and its higher/ lower degree of manipulation or no manipulation). Other contextual factors need to be considered in this respect. Given our tolerance to image manipulation in advertising, for example, and the likely rejection of such strategy when translating a film, it will be important to investigate the correlation between translation strategies, translators’/viewers’ attitudes and issues such as genre, authorship, prestige (of the director and film), function of the text, and audience expectations.

4. Conclusion: A joint research agenda With seeing the world as a multimodally orchestrated meaningful reality comes the realisation that actual translation is infinitesimal compared to the signs presented to us or surrounding us that are not translated. If this has always been the case without being perceived as an issue, then why do we need to further research on “multimodality and translation”? Without wanting to frame the issue of “equivalence” in terms of faithfulness, we believe that questions need to be asked regarding the profile of the texts target audiences have access to, but also how this might be indirectly reinforcing power imbalances. After all, due to their transnational prestige and wide distribution, Western sign-­making practices are possibly more easily interpreted (and their underlying values and ideologies absorbed), as in the case of The Big Bang Theory examined earlier. This might lead to standardisation of meaning-­making practices in favour of Western practices, while pushing divergent practices (and the “other” behind them) to the periphery, which we have more difficulty accessing and interpreting. We hope to have highlighted the importance of considering translation beyond the verbal but, in doing so, we know how many unanswered questions we are left with. In TS, an increasing number of works analyse translation and localisation of multimodal products (O’Hagan and Mangiron 2013; Dicerto 2018), yet the cases examined are still very specific and isolated, making it difficult to compose a wider

84  Elisabetta Adami and Sara Ramos Pinto

picture of generalizable significance for TS as a whole and day-­to-­day translation practice. Useful theoretical discussions have so far made sense of the state-­of-­the-­ art on the issue (O’Sullivan and Jeffcote 2013; Pérez-­González 2014; Gambier and Ramos Pinto 2016; Dicerto 2018), yet the lack of systematic investigations on the subject makes it difficult to derive broader frames/approaches that enable us to address the relation between multimodality and translation more holistically. We strongly believe that it is no longer possible to move forward without a more robust theoretical framework grounded on relevant empirical data yet to be collated. In this concluding section we sketch the work needed in semiotic and translation research to further our understanding of the implications of multimodality for translation, to ground empirically the decisions to be made in translation practice, as well as to foresee their possible implications. We present the types of research questions and the lines and areas of enquiry, along with the methodological and theoretical integrations required to attempt to answer them.

4.1  Research questions Faced with a semiotic landscape of transnationally circulating untranslated signs (except language), the semiotician asks “how do people make meaning out of this?” While issues of culture specificity in meaning-­making have been so far disregarded, social semiotics needs to start also asking “who shares (some extent of) semiotic knowledge about the uses of specific resources, and for the meaning made through their relations?”; this means asking “what is shared/non-­shared culture in a given semiotic landscape?”, “how do the ways in which different resources and signs circulate influence the ways in which they are interpreted?”, and “how does the participation in different affinity spaces/communities influence an individual’s meaning-­making of nonverbal resources?” In conjunction with insights deriving from addressing the aforementionedquestions, TS needs firstly to provide descriptions and explanations about past and current translation practice (following a historical and cultural approach), asking questions such as “how have multimodal products and nonverbal elements within them been approached in translation?” and “how has that mediated the circulation and reception of translated multimodal products as well as our understanding of the other(s)?” Secondly, and possibly grounded empirically on findings from all of these questions, TS needs to seek ways to provide support for informed decisions in translation practice, thus asking questions such as “how do we identify the elements in need of translation considering that meaning is multimodally constructed?”, “how do people interpret translated multimodal products?”, and “what is the impact of specific translation strategies in cross-­relation with factors such as modes, media, genre, domain, audience, and purpose?” We do not want to reduce the aims of social semiotics and TS research to a mere provision of indications for translation practice and training. Providing answers to these questions can further understanding into the social dynamics of meaning-­ making and into the very nature of translation, while supporting better-­informed

Meaning-­(re)making  85

decisions in translation practice and kickstarting a reflection on how translation training could develop to answer today’s challenges.

4.2  Lines and areas of research To start addressing these questions, a first broad line of enquiry involves investigating how shared/non-­shared semiotic knowledge distributes across populations and individuals participating in different semiotic spaces. A second line of enquiry involves investigating how humans make meaning the first time they engage with specific signs.

4.2.1 Distribution of semiotic knowledge: modes, media, genres, and domains The first line of enquiry requires examining sign-­making and meaning-­making practices in (at least) three intertwined areas. One area involves investigation of specific semiotic modes to understand their historical development and the extent of codification and dynamics of transnational circulation (which may be different for different modes). A second area involves investigating meaning produced multimodally in/across different media, as the way we make meaning may differ between a printed advertisement and a social media meme due to media-­specific expectations regarding purpose, kinds of authorship, context, etc. A third area needs to examine sign-­and meaning-­making practices in specific genres and domains, with their own specific communities, how these differ and/or distribute transnationally (the resources of image have not undergone the same type of codification in, e.g., technical drawing, cartoons, and maps; some theatre traditions, like the Japanese one, have had a different transnational reach and higher level of national codification than others), as well as how they influence each other across genres and domains. A better understanding of meaning-­making practices and how these circulate will assist translators in their own interpretation of the source text and evaluation of what needs to be translated. It will also assist TS in mapping the development/circulation of translated multimodal products and examination of translators’ choices. The intertwined areas of investigation (mode, media, genres, and domains) on shared/non-­shared semiotic knowledge need to be (i) situated in and across time/ space and (ii) mapped onto and across linguistic communities, for multimodal texts that also involve (and require translation of ) language. Investigation through time (which would give way to historical social semiotic research) needs to include multiple timescales (Lemke 2000, 2009) and be considered in its inseparable relation with space (as chronotopes, Bakhtin 1981). This involves a radical rethinking of notions of boundaries, which may often not coincide with language/national borders. Especially in today’s globalised and technologically connected societies, space needs to be less geographically (and linguistically) conceived and more socially defined (following conceptions of space and place that stem from Lefebvre 1984; for a review, Jaworski and Thurlow 2010). The mapping of semiotic knowledge

86  Elisabetta Adami and Sara Ramos Pinto

onto (more or less linguistically homogeneous) social groups can and needs to lead to a re-­definition of social group in a transnationally connected world (unevenly distributed between hegemonic and minoritised cultural flows). Given that individuals participate in multiple and multilingual spaces/communities/groups (as in the example of anime fans mentioned earlier), each with specific semiotic practices, investigation should test the advantages of adopting notions centred on spaces (as in Gee’s 2005 affinity spaces) rather than those centred on groups/communities, possibly also contributing to a shift in perspective within TS from the problematic notion of source/target audiences to that of source/target spaces (of shared semiotic knowledge). A perspective on spaces where texts and semiotic practices and knowledge circulate would avoid associations to people’s membership to communities (which trigger epistemologically dangerous national/local labelling, e.g., the Japanese, the Germans, etc.), while focusing on practices (rather than people’s belonging) and embracing more immediately the fluid character of their circulation in, out, across, and through different spaces.

4.2.2  Principles of semiosis Investigation of how modes have developed specific uses, how signs circulate through different media and in specific genres and domains, how their meanings change across time/space, and how shared/non-­shared semiotic knowledge distributes across affinity spaces, social groups, and linguistic communities cannot be done through fixed categorisations, which would risk producing artificial inventories/ taxonomies. These investigations need to be dynamically conceived; individuals’ semiotic repertoires change and vary constantly as they encounter new signs/representations and engage in new interactions; in other terms, semiotic repertoires are relational, adaptive, and flexible (see applications of Complexity Theory to language development in Cameron and Larsen-­Freeman 2007; Larsen-­Freeman 2015). Consequently, rather than compiling a set of formal tools of reference for, e.g., visual signs, a second line of enquiry needs to account for the dynamic complexity of human semiosis; thus, the mapping of shared/non-­shared semiotic knowledge needs to be paired with research focused on understanding general/common meaning-­making principles and strategies. If shared semiotic knowledge enables common ground of interpretation between sign-­and meaning-­maker (e.g., interpreting the floppy disk button as the “save” functionality), then in-­depth understanding of common principles of semiosis can enable the formulation of informed hypotheses on how a sign will be interpreted when it is engaged with for the first time (as when encountering a new symbol for the button of a given software functionality, or a new filmic effect), thus further supporting informed decisions in translation practice. This can be done by empirically testing current theories of semiosis, such as Kress’s (1993, 2010) motivated sign (cf. Section 1), as well as by testing against the whole semiosis current theories on meaning-­making developed for language, such as Relevance Theory (Sperber and Wilson 1986) and Conceptual Metaphor (Lakoff and Johnson 1980).

Meaning-­(re)making  87

These two lines of enquiry, grounded in and across time/space, and situated in and across different modes, media, and genres/domains, are the first necessary steps for an empirically informed formulation of hypotheses on how a given resource will be interpreted by members of a given audience (or by appearing in a given affinity space), whether any mediation will be needed, and what kind of strategy would be most apt.

4.3  Methodological developments The research that we envisage demands methodological development and integration to encompass exploration of (i) multimodal products and how these circulate, (ii) the practices of multimodal text production and translation, and the discourses attached to them, as well as (iii) practices of (re)interpretation and re-­signification. Investigating the research questions in the lines and areas of enquiry sketched here requires extensive descriptive studies on actual multimodal products. These demand new tools of multimodal analysis capable not only of integrating the investigation of all modes in equal terms, as well as the intermodal relations and their translation, but also of enabling quantitative analysis of larger datasets that can offer generalizable results. Traditional social semiotic multimodal analysis has developed tools for fine-­grained investigation of small samples of multimodal texts. These findings can hardly be combined with those from textual corpora analysis (which takes advantage of quantitative data analysis), thus reinforcing even further the unequal treatment of nonverbal resources vs. language. Development of multimodal corpus analysis requires implementation of software tools for data mining of nonverbal resources and integration of extant technologies for visual recognition. Alongside larger-­scale examinations of multimodal products and the identification of regularities in sign-­making and its translation, it will be essential to investigate both ends of production and what is commonly termed as “reception” in TS. We need methodological integration to examine the discourses and practices of text production and translation as well as the audiences’ interpretations and the impact of translation strategies on such interpretations. By cross-­relating the regularities found in the analysis of multimodal products with contextual factors, we will be able to examine the social, economic, and cultural factors promoting specific multimodal sign-­making and translation practices, as well as to map their development, circulation, and reception. This will require the undertaking of large sociological investigations that revisit Bourdieu’s (1986) methods and variables for mapping “cultural capital” against the contemporary socio-­semiotic landscape (for a problematisation of the concept, Adami 2018), as well as extensive reception studies focused on collecting data on meaning-­making processes across different variables such as genre, purpose, and type of audience. Data collection methods range from interviews, surveys, and questionnaires to eye-­tracking and memory recall protocols (for an overview, Pérez-­González 2014). Three further methodological issues need to be addressed. Firstly, most data-­collection methods and means of academic dissemination are still heavily

88  Elisabetta Adami and Sara Ramos Pinto

logocentric, forcing both research participants and researchers to describe multimodally produced meaning almost exclusively through language (be it speech for participants’ interviews in reception studies or writing for both questionnaires and for academic publications). The exploration of meaning of nonverbal resources requires us to develop ways to elicit participants’ feedback beyond verbal output to avoid possible risks of forcing meaning into hardly apt linguistic categorisations (which, moreover, require certain levels of literacy and articulation). Some methodological attempts in this sense are being developed in social semiotics (as when asking participants to draw rather than to tell their meanings, e.g., Kress 2010) and in sensory research (e.g., asking participants to gesture or make shapes to express what they taste).7 These methods need to be systematically developed and adopted also in studies concerned with meaning of translated multimodal products. Parallel to forms of nonverbal data collection, we need new forms of multimodal dissemination of findings; research in “multimodality and translation” could contribute further to push the academic publishing system to promote (recognition of) forms of dissemination such as visual and video essays, as well as online publishing that exploits the hypertextual and multimedia affordances of the web (see the journal Kairos, http://kairos.technorhetoric.net/). Secondly, even if very much needed to fill a gap in multimodal research, large-­ scale quantitative investigations are not enough to address the research questions mentioned earlier. Big data analysis can incur the risk of overgeneralising differences in meaning-­making practices onto macro-­categories, such as assumed “national cultures” or “linguistic communities,” while screening out crucial micro-­level variabilities. To explore how variables such as individuals’ personal trajectories and participation in multiple semiotic spaces influence their meaning-­making of nonverbal resources (and in their relation with language), there is a need for in-­depth, fine-­ grained qualitative research. This requires integration with ethnographic approaches, which is already underway in social semiotic multimodal analysis (e.g., Dicks et al. 2011, although not focusing on “culture/semiotic knowledge”), and could constitute a methodological innovation for TS, leading the way to an ethnography of translation. Findings at a more qualitative level could inform the testing of hypotheses at the quantitative level, while the latter results could be further probed qualitatively, through iterative cycles of analysis. Lastly, the type of multi-­layered examination proposed here will only be possible if we embrace large-­team (quantitative and qualitative) examinations involving both professional and “lay” designers and producers of multimodal products, as well as audiences, alongside researchers in social semiotics, TS, sociology, ethnography, cognitive linguistics, and statistics, among others. Research could thus explore participatory methods more extensively. Analogous to the role of citizen sociolinguists advocated by Rymes (2014), both semiotic and translation research should consider the value of today’s distributed knowledge and start conceiving of contributions outside of academia in the form of “citizen socio-­semioticians” and “citizen translation scholars.” We all reflect and meta-­comment not only on our and other people’s use of language, but also on images, clothing, font, music, architecture, and

Meaning-­(re)making  89

all sign-­making practices that we encounter, as well as on actual translations and how we think these could better fulfill our needs. Given the breadth and width of the phenomenon to be investigated and its research directions, knowledge and resources need necessarily be drawn also from outside the confines of academia.

4.3  Theoretical integration Such a broad and articulated research enterprise needs necessarily to cross disciplinary boundaries, not only between social semiotics and TS. We list here four transdisciplinary directions: •

•

•

•

Integration is needed with linguistics in several respects. To the suggestions mentioned throughout the chapter, we add the potential contribution of studies on relevance (in the tradition of Sperber and Wilson 1986, yet re-­defined in a co-­text rather than context approach), narratology salience (Bortolussi and Dixon 2003), and cognitive load and processing (Kalyuga 2012), as well as a cultural mapping and comparative expansion of extant cognitive approaches to multimodality (e.g., Forceville and Urios-­Aparisi 2009). Knowledge on the development and circulation of specific semiotic resources in different contexts can be drawn from specialised fields, such as visual communication, media studies, graphic design, musicology, art history, and film studies. Transdisciplinary crossing is required also with studies on culture in anthropology, sociology, and intercultural communication, to test and re-­define different notions of culture against a co-­text rather than context approach. In this, social semiotics could use extant definitions of culture to verify variation in patterns of semiosis, while studies on culture could benefit from a semiotic perspective to redefine notions of culture (in a co-­text rather than context approach). This transdisciplinary undertaking could also facilitate reconceptualisations of target audience/space in TS. Because of the multiple dimensions of human semiosis, which involves not only social but also psychological, biological, sensory/perceptual, and material aspects and variables, further transdisciplinary engagement is required with other disciplines, such as psychology, sensory ethnography, material semiotics, and cognitive neuroscience, each being equipped to investigate one specific aspect, while these need to be considered in their complex intertwining.

The joint semiotic/translation research enterprise sketched here is only preliminary; it requires refinement and further articulation, possible only once investigation has started; yet, as is, it is already indicative of its wide, broad, and extensively ambitious scope and reach. The whole enterprise is not exempt of risks; we would like to conclude by mentioning a caveat, which has started to concern us while discussing the possible implications of such an investigation. A mapping of shared/ non-­ shared semiotic knowledge onto social groups risks producing selective

90  Elisabetta Adami and Sara Ramos Pinto

descriptions, and hence prescriptions and regulations on uses, which may not only constrain individuals’ agency in sign-­making but also contribute to exacerbating the divide between hegemonic and minoritised cultural practices (impacting on the people associated with them). Analogously, description of multimodal translation practices and formulation of possible (inter-­and intra-­modal) translation strategies may risk producing prescriptions and promoting practices that favour a supposedly easier-­to-­interpret domestication of nonverbal resources, thus risking to contribute to separating communities and constraining their meaning-­making possibilities as well as their exposure to diversity in sign-­making. Research in sociolinguistics and applied linguistics (e.g., García 2009; García and Wei 2014; Canagarajah 2017) has provided ample evidence of the negative effects of national codification, standardisation, and prescription onto language use, pointing to their role in (re)producing social inequalities. Drawing on the insights of linguistic research on these themes, the investigative enterprise we are proposing needs to be entirely innovative and thoughtful of the possible implications, and not merely reproduce or adapt methods used for language in the past.

Notes 1 The history of the nosebleed to signify sexual arousal in anime is uncertain, and multiple explanations have been hypothesised; these range from tracing the use back to single authors (like manga artist Yasuji Tanioka in the 1970s), to unverified popular sayings in the Japanese tradition used to deter youths from sexual situations, up to identifying the nosebleed as a taboo-­avoidance metaphor for male ejaculation (later extended also to female characters in anime); for a discussion, see https://kotaku.com/sexual-­arousal-­doesnt-­cause-­bloody-­ noses-­says-­medical-­5953124 (Accessed 30 August 2018); an online search of ‘anime nosebleed’ (and their equivalent Japanese hanaji or 鼻血) generates dozens of different sources, discussions, and hypotheses. The very range of these substantiates the social semiotic postulate on meaning-­making/interpretation as an individual’s hypothesis on the motivation underpinning the association of a form/signifier to a meaning/signified, on the basis of the combined materiality and past uses of a given resource. 2 Social semiotics avoids the label “nonverbal” for resources other than speech and writing, because of its implied subordination of these resources to language, and because the stem “verbal” assumes that writing and speech can be considered as one mode (as the label “language” supposes). We use it here as a shorthand way to label all resources that, unlike speech and writing, have been considered para-­or extra-­linguistic and hence context, in traditional linguistic research and TS. 3 This has applied also to its non-­citizen residents (as shown in current controversies on migrants’ language literacy) and to colonised peoples (e.g., Canagarajah 2017). 4 This is proved, through counter-­evidence, by the fluidity of language use, meaning, and perception among speakers of oral communities (see, for example, Goodchild and Weidl 2018). 5 Meme available from: https://me.me/i/i-­showed-­my-­12-­year-­old-­son-­an-­old-­floppy-­ 3250837 (Accessed 22 November 2018). 6 We isolate one single sign and shot for exemplifying purposes; obviously, the more audiences are exposed to intertextual relations throughout the series, the more they will construct (shared) semiotic knowledge on the signs of “geeky-­ness,” as happens in all exposure of foreignising translations. 7 On sensory research, see work conducted by CenSes, The Centre for the Study of the Senses: https://philosophy.sas.ac.uk/centres/censes.

Meaning-­(re)making  91

References Adami, Elisabetta. 2017. “Multimodality and Superdiversity: Evidence for a Research Agenda.” Tilburg Papers in Culture Studies 177: 1–28. ———. 2018. “Shaping Public Spaces from Below: The Vernacular Semiotics of Leeds Kirkgate Market.” Social Semiotics 11 (October): 1–25. doi:10.1080/10350330.2018.1531515. Bakhtin, Mikhail. 1981. The Dialogic Imagination: Four Essays. Austin: University of Texas Press. doi:10.2307/2068977. Bezemer, Jeff, and Gunther Kress. 2008. “Writing in Multimodal Texts: A Social Semiotic Account of Designs for Learning.” Written Communication 25: 166–95. Bordwell, David, and Kristin Thompson. 1979. Film Art: An Introduction. New York: McGraw-­Hill. Bortolussi, Marisa, and Peter Dixon. 2003. Psychonarratology: Foundations for the Empirical Study of Literary Response. Cambridge: Cambridge University Press. Bourdieu, Pierre. 1986. Distinction: A Social Critique of the Judgement of Taste. London and New York: Routledge. Cameron, Lynne, and Diane Larsen-­Freeman. 2007. Complex Systems and Applied Linguistics. Oxford: Oxford University Press. Canagarajah, Suresh. 2017. The Routledge Handbook of Migration and Language. London and New York: Routledge. Cattrysse, Patrick. 1992. “Film (Adaptation) as Translation: Some Methodological Proposals.” Target 4 (1): 53–70. Chaume, Frederic. 2016. “Audiovisual Translation Trends: Growing Diversity, Choice, and Enhanced Localization.” In Media Across Borders: Localizing TV, Film and Video Games, edited by Andrea Esser, Miguel Angel Bernal-­Merino, and Ian Robert Smith, 68–84. London and New York: Routledge. Chesterman, Andrew. 2005. “Consilience in Translation Studies.” Revista Canaria de Estudios Ingleses 51 (Novembre): 19–32. ———. 2007. “Bridge Concepts in Translation Sociology.” In Constructing a Sociology of Translation, edited by Michaela Wolf and Alexandra Fukari, 171–83. Amsterdam and Philadelphia: John Benjamins. Chiaro, Delia. 2014. “The Eyes and Ears of the Beholder? Translation, Humour, and Perception.” In Media and Translation: An Interdisciplinary Approach, edited by Dror Abend-­David, 197–220. London and New York: Bloomsbury. Dicerto, Sara. 2018. Multimodal Pragmatics and Translation: A New Model for Source Text Analysis. London: Palgrave Macmillan. Dicks, Bella, Rosie Flewitt, Lesley Lancaster, and Kate Pahl. 2011. “Multimodality and Ethnography: Working at the Intersection.” Qualitative Research 11 (3): 227–37. Esslink, Bert. 2000. A Practical Guide to Localisation. Amsterdam and New York: John Benjamins. Forceville, Charles, and Eduardo Urios-­Aparisi. 2009. Multimodal Metaphor: Journal of Pragmatics,Vol. 42. Berlin: Mouton De Gruyter. Gambier, Yves. 2003. “Screen Transadaptation: Assessment and Reception.” The Translator 9 (2): 171–89. ———. 2013. “Moves Towards Multimodality and Language Representation in Movies.” In Reading and Intersemiosis in Multimedia: Proceedings of the 3rd ICOM Pavia, May 2006, edited by Elena Montagna, 1–10. Pavia: Como. Gambier, Yves, and Sara Ramos Pinto. 2016. “Special Issue on Audiovisual Translation: Theoretical and Methodological Challenges.” Target 28 (2). García, Ofelia. 2009. “Education, Multilingualism and Translanguaging in the 21st Century.” In Multilingual Education for Social Justice: Globalising the Local, edited by Ajit Mohanty,

92  Elisabetta Adami and Sara Ramos Pinto

Minati Panda, Robert Phillipson, and Tove Skutnabb-­Kangas, 140–58. New Dehli: Orient Blackswan. García, Ofelia, and Li Wei. 2014. Translanguaging: Language, Bilingualism and Education. New York: Palgrave Macmillan. Gee, James Paul. 2005. “Semiotic Social Spaces and Affinity Spaces: From The Age of Mythology to Today’s Schools.” In Beyond Communities of Practice: Language, Power and Social Context, edited by David Barton and Karin Tusting, 214–32. Cambridge: Cambridge University Press. Gibson, James. 1979. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin. Goodchild, Samantha, and Miriam Weidl. 2018. “Translanguaging Practices in the Casamance, Senegal: Similar but Different: Two Case Studies.” In Making Signs,Translanguaging Ethnographies: Exploring Urban, Rural, and Educational Spaces, edited by Ari Sherris and Elisabetta Adami, 133–51. Clevedon: Multilingual Matters. Halliday, Michael. 1978. Language as Social Semiotic: The Social Interpretation of Language and Meaning. London: Arnold. Hawkins, Margaret R. 2018. “Transmodalities and Transnational Encounters: Fostering Critical Cosmopolitan Relations.” Applied Linguistics 39 (1): 55–77. Hodge, Robert, and Gunther Kress. 1988. Social Semiotics. Cambridge: Polity. Holliday, Adrian. 1999. “Small Cultures.” Applied Linguistics 20 (2): 237–64. Jaworski, Adam, and Crispin Thurlow. 2010. “Introducing Semiotic Landscape.” In Semiotic Landscapes: Language, Image, Space, edited by Adam Jaworski and Crispin Thurlow, 1–41. London: Continuum. Jiménez-­Crespo, Miguel. 2013. Translation and Web Localisation. London and New York: Routledge. Kalyuga, Slava. 2012. “Instructional Benefits of Spoken Words: A Review of Cognitive Load Factors.” Educational Research Review 7 (2): 145–59. Kovačič, Irena. 1995. “Reception of Subtitles: The Non-­Existent Ideal Viewer.” Translatio (FIT Newsletter) 14 (3–4): 376–83. Kress, Gunther. 1993. “Against Arbitrariness:The Social Production of the Sign as a Foundational Issue in Critical Discourse Analysis.” Discourse and Society 4 (2): 169–91. ———. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. London and New York: Routledge. Kress, Gunther, and Theo van Leeuwen. 1996. Reading Images:The Grammar of Visual Design. London and New York: Routledge. Lakoff, George, and Mark Johnson. 1980. Metaphors We Live By. Chicago: University of Chicago Press. Larsen-­Freeman, Diane. 2015. “Complexity Theory.” In Theories of Second Language Acquisition, edited by Bill Van Patten and Jessica Williams, 2nd ed., 227–44. London and New York: Routledge. Lefebvre, Henri. 1984. The Production of Space. Oxford: Blackwell. Lemke, Jay. 2000. “Across the Scales of Time: Artifaces, Activities, and Meanings in Ecosocial Systems.” Mind, Culture and Activity 7 (4): 273–90. ———. 2009. “Multimodality, Identity, and Time.” In The Routledge Handbook of Multimodal Analysis, edited by Carey Jewitt, 140–50. London and New York: Routledge. Leppihalme, Ritva. 1997. Culture Bumps: An Empirical Approach to the Translation of Allusions. Clevedon: Multilingual Matters. New London Group. 1996. “A Pedagogy of Multiliteracies: Designing Social Futures.” Harvard Educational Review 66: 60–92.

Meaning-­(re)making  93

O’Hagan, Minako, and Carmen Mangiron. 2013. Game Localization: Translating for the Global Digital Entertainment Industry. Amsterdam and New York: John Benjamins. O’Sullivan, Carol, and Caterina Jeffcote. 2013. “Special Issue on Translating Multimodalities.” Journal of Specialized Translation 20. Pérez-­González, Luis. 2014. “Multimodality in Translation and Interpreting Studies.” In A Companion to Translation Studies, edited by Sandra Bermann and Catherine Porter, 119–31. Chichester: Wiley-­Blackwell. Ramos Pinto, Sara. 2018. “Film, Dialects and Subtitles: An Analytical Framework for the Study of Non-­Standard Varieties in Subtitling.” The Translator 24 (1): 17–34. Remael, Aline. 2001. “Some Thoughts on the Study of Multimodal and Multimedia Translation.” In (Multi)Media Translation: Concepts, Practices and Research, edited by Yves Gambier and Henrik Gottlieb, 13–22. Amsterdam and New York: John Benjamins. Rymes, Betsy. 2014. Communicating Beyond Language: Everyday Encounters with Diversity. London and New York: Routledge. Sperber, Dan, and Deirdre Wilson. 1986. Relevance: Communication and Cognition. Oxford: Blackwell. Stöckl, Hartmut. 2004. “In Between Modes: Language and Image in Printed Media.” In Perspectives on Multimodality, edited by Eija Ventola, Cassily Charles, and Martin Kaltenbacher, 9–30. Amsterdam and New York: John Benjamins. van Leeuwen, Theo. 2005. Introducing Social Semiotics. London and New York: Routledge. Zanettin, Federico. 2014. “Comics in Translation: An Overview.” In Comics in Translation, edited by Federico Zanettin, 1–32. London and New York: Routledge. Zitawi, Jehan Ibrahim. 2014. “Disney Comics in the Arab Culture(s). A Pragmatic Perspective.” In Comics in Translation, edited by Federico Zanettin, 270–306. London and New York: Routledge.

4 FROM THE “CINEMA OF ATTRACTIONS” TO DANMU A multimodal-­theory analysis of changing subtitling aesthetics across media cultures Luis Pérez-­González

1. Introduction Multimodal artefacts, including audiovisual texts, result from the interplay between different types of co-­operating signifying resources that have to be “described and describable together” (Kress and Ogborn 1998, quoted in Iedema 2003, 39), not least when accounting for the transformations such texts undergo through translation. But while terms like “multimodality” or “multimodal theory” feature ever more frequently in translation studies research, the interface between both disciplinary domains remains woefully under-­explored. As Christopher Taylor has noted, multimodality theorists have “not as yet focused on questions of translation” (2016, 222), while translation scholars have not yet begun to make use in a systematic and sustained fashion of the concepts and methods that multimodality has to offer, despite the widely held perception that the latter provides a robust inter-­ and transdisciplinary platform to support research across the humanities. As far as audiovisual translation research is concerned, this divide has been exacerbated by the entrenched prevalence of “monomodality” – understood as the dominance of one signifying constituent, typically written language, over other types of meaning-­ making resources or modes used in a text – in what have been traditionally perceived as the most prestigious forms of cultural or artistic expression and “[t]he specialised theoretical and critical disciplines which developed to speak of these arts” (Kress and van Leeuwen 2001, 1). Until relatively recently, audiovisual texts and their translations have not featured among such artistic manifestations or in the discourses that the latter mobilise. A robust challenge to the dominance of monomodality, however, has been mounted by the scale and depth of recent changes to the medial environment in which translations are produced and received. Over the last three decades, technological advances triggered by the shift from the print to the screen and

From the “cinema of attractions” to danmu  95

then to digital culture have been instrumental for the emergence of new textualities where written language routinely interacts with still/moving images and/or sound in new and productive ways – whether within physical artefacts such as magazines or textbooks, or in the context of hypertextual assemblages. The impact of these transformations in “materiality and its cognates, mediality and technicity” (Littau 2016a, 82) on the semiotic make-­up of digital texts has not gone unnoticed by translation scholars. Their heightened awareness of the materiality of communication – encompassing “all those phenomena and conditions that contribute to the production of meaning, without being meaning themselves” (Gumbrecht 2004, 8, quoted in Littau 2016a, 83) – has fostered deeper engagement with the dialectic between media and semiotic resources, and has placed the material history of translation under renewed scrutiny (Littau 2016b). As Elisabetta Adami and Sara Ramos Pinto emphasise in their chapter for this volume (Chapter 3), these developments have occurred in different ways in different cultures during a period of increasing globalisation. In this context, calls for “an integrated approach to translation” that gives careful consideration to “the physical support (stone, papyrus, CD-­ROM), the means of transmission (manuscript, printing, digital communication),” and “how translations are carried through societies over time by particular groups” (Cronin 2003, 29) are congruous with efforts undertaken in other disciplinary domains to develop conceptual and methodological tools for the study of communicative practices in all their semiotic complexity (Iedema 2003). Against the backdrop of growing engagement with the study of multimodal texts, this chapter sets out to chart the changing ontological contribution that subtitles have made to the multimodal fabric of audiovisual texts produced since the silent film era to the present. A number of multimodal issues associated with subtitling have been reported in previous publications that do not explicitly associate themselves with multimodality studies (see Pérez-­González 2020 for a detailed survey), yet a growing body of subtitling scholarship that subscribes explicitly to multimodal theory has explored the disciplinary connections between multimodality and audiovisual translation in more depth (Remael 2001; Taylor 2003; Gambier 2006; Pérez-­González 2007; Mubenga 2009; O’Sullivan 2011; Desilla 2012; Taylor 2014; Gambier and Ramos Pinto 2016). This chapter offers a novel approach to the study of the multimodal dimension of subtiling practices, examining the organic enmeshment of subtitles within the overall semiotics of audiovisual content across successive media cultures, moulded by evolving configurations of technology, power, ontological status, and social practices. The chapter aims to yield an in-­depth understanding of the implications that the shift from the dominant narrational regime of Western modernity to postmodern aesthetics emerging in the context of digital culture has for multimodal textualities and their translation. This study is therefore not based on the analysis of small-­scale translation shifts in a multimodal context. Instead, it engages with wider theoretical and programmatic debates in translation, multimodal, and cultural studies to inform future reflective practice at this disciplinary interface.

96 

Luis Pérez-­González

The chapter is structured as a critique of the changing role of subtitling in the media landscape, charting the gradual emancipation of this modality of translation from the dictates of the film industry as it comes to serve the more democratic forms of spectatorial media “prosumption” associated with digital culture. Gunther Kress and Theo van Leeuwen’s distinction between “recording” and “synthesising technologies” provides the conceptual network required to trace the shift from an “ontology of referentiality,” based on the hegemonic narrational regime, to the “ontology of deconstruction” that lies at the heart of the digital media ecology, shaped by variable degrees of convergence between industrial and amateur practices (Kress and van Leeuwen 2006, 217). Developments in the digital arena, as instantiated by Chinese danmu, are presented as contemporary instances of “representation-­ as-­design” driven by rhetorical choices (Kress and van Leeuwen 2006, 219) and analysed in terms of the affordances of “semiotic software” (Djonov and van Leeuwen 2017) and the social semiotic practices that the latter enables. Ultimately, this study shows how, where ordinary rhetors have gained greater visibility and agency, their performance of citizenship involves the deconstruction of representation by exposing the cultural and social make-­up of multimodal ensembles.

2. Subtitling in the era of recording technologies As is also the case with users of other multimodal texts (for instance, see Marcus Tomalin’s discussion of the livre d’artiste in his chapter for this volume, Chapter 6), mainstream film audiences must make sense of meaning distributed across various semiotic systems. The range of signifying means deployed by film creators to encode multiple strands of acoustic and visual meaning effectively constitute “a single unified gestalt in perception” (Stöckl 2004, 16). Presented with these unified semiotic entities, film viewers have to make connections across different types of signifying resources in a routinised, subconscious manner, drawing on their accrued spectatorial experience to decode conventionalised ensembles of sign types. The demands originally placed on viewers have grown as film semiotics began broadening its repertoire of meaning-­making resources as early as in the pre-­sound film era. The relentless drive to heighten the appeal of motion pictures as commodities led silent film creators to attempt to convey mute diegetic speech to their audiences in various ways, including, but not limited to, film editing techniques and the use of live narrators, “speaking actors,” title cards, and intertitles (Dwyer 2005; Pérez-­ González 2014a; O’Sullivan and Cornu 2018). The advent of sound in the late 1920s further cemented the status of film as the first modern audiovisual form of entertainment and paved the way for the development of subtitling and dubbing in a bid to preserve and increase the commercial flow of films across linguacultures. Subtitling is widely held to disrupt the multimodal configuration of the original artefact, as conceived by filmmakers, for the ex post-­facto incorporation of snippets of text to a finished film calls for additional intermodal connections that viewers of the original ensemble were not required to draw. While commercial subtitles delivering translations of the original dialogue do

From the “cinema of attractions” to danmu  97

not generally influence either the “pacing and rhythm” of the film or its “intellectual and emotional content” (MacDougall 1998, 168), the need to process written language superimposed on the image represents an additional cognitive effort that may detract from the enjoyment of audiences in dubbing-­dominated markets. The impact of dubbing conventions on the co-­deployment of semiotic resources in films will not be pursued further in this chapter due to space constraints. Although the partnership between film and subtitles spans approximately one century, we are only beginning to gain reliable insights into the implications that the superimposition of snippets of written text on images has for the processing of subtitled films. The burgeoning body of eye-­tracking research on the behaviour of viewers’ gaze shows that the on-­screen mobilisation of subtitles always draws attention from viewers, even when audience members are able to understand the dialogue they are presented with (d’Ydewalle and De Bruycker 2007). After reading the subtitles (Jensema et al. 2000), gaze trajectories generally alternate between the subtitles and the images they are inscribed on, as viewers adjust their processing strategies according to the genre they are watching (Perego et al. 2010). Most viewers spend more time reading the subtitles than looking at the images, although the fixations – understood as the pauses between eye movements, during which the viewer’s eyes remain static over an area of interest – on the latter are normally longer (Perego et al. 2010; Kruger 2018). Effectively, once viewers have read the subtitles in full, they tend to focus their attention on key areas of interest – primarily faces, when these are present – for as long as possible. The evidence emerging from eye-­tracking research thus confirms that viewers of subtitled films have less time to explore visual semiotic resources, although audiences in subtitling-­dominated audiovisual markets are able to watch subtitled programmes relatively effortlessly, as they are regularly exposed to this specific multimodal configuration (d’Ydewalle and De Bruycker 2007). Crucially, this incipient body of evidence was not yet available during the 1930s and 1940s, at the time when the industry was developing the set of subtitling practices that, as acknowledged by film scholars, have remained virtually unchanged ever since (James 2001). The conventions in question seek to achieve full synchrony between the duration of successive blocks of diegetic speech and the on-­ screen display of the subtitle(s) conveying their meaning into the target language, assuming standard (albeit adjustable) sets of reading conditions associated with different viewer profiles. Under these “spatio-­temporal constraints” imposed by the film medium, spoken dialogue must be substantially condensed to fit into compact, tightly worded subtitles, which have been shown to have a detrimental impact on the viewers’ perception of characterisation, humour, irony, and other culture-­ specific aspects of subtitled film (Mason 2001; Remael 2003; Desilla 2012). Drawing on Stöckl’s (2004) conceptualisation of multimodal resources, to be elaborated on later, the process of semiotic transfer known as industrial subtitling therefore revolves almost exclusively around two “medial variants” (written and spoken) of the same “mode” (language). Except for those instances where the decoding of specific knowledge (see featured nonverbal semiotic resources requires culture-­

98 

Luis Pérez-­González

example 6.1 in Pérez-­González 2014a, 188–90), it is generally assumed that viewers will manage to activate any connections across different types of semiotics that the subtitled film requires, just as the original film audience members did. Critical theory and film studies scholars with an interest in the genealogy of subtitling practices, however, have taken the view that industrial subtitling practices are not, strictly speaking, the necessary product of spatio-­temporal constraints associated with the film medium. Instead, the cult of synchrony underpinning those subtitling conventions is argued to be consistent with other semiotic developments in the production of motion pictures, not least the evolution of early presentational films – i.e., “the films of the cinema of attractions” where “theatrical display dominates over narrative absorption” (Grochowski 2007, 56) – into representational ones, which seek to create a self-­contained diegetic world driven by narrative motivation allowing viewers to suspend their disbelief. In addition to the deployment of sophisticated editing and montage practices, the construction of this self-­enclosed fictional world is heavily contingent on the use of synchronous diegetic sound because it provides an allegedly “unmediated access to [the diegetic] reality” (Minh-­ha 2005, 129). Synchronous diegetic sound allows producers to shift viewers’ attention away from the tools and relations of production – i.e., the spaces between image, sound, and text that can be seen in the films of the cinema of attractions – and minimises the potential for subjective spectatorial experiences. So while sound synchrony would, on the face of it, aim to promote an aesthetic of objectivity, the centrality of the diegetic in modern film production ultimately signals the industry’s interest in spreading hegemonic commercial discourses that work through linear narratives to control film reception. Drawing on Gunning (1986) and Musser (2006), Grochowski explores the financial and political drivers behind the expansion of the Hollywood studios, gauging the extent of their impact on the consolidation of the narrational regime. For Grochowski, the synchrony-­driven switch from the presentational to the narrational was the outcome of a “struggle between producers of motion content and exhibitors (where the production of narrative films becomes merely a strategy, a “tool” to use by the producer to control exhibition)” (2007, 56). Ultimately, sound synchrony, and the shift from the presentational to the narrational paradigm that it brought about, enabled the US film industry to impose a taste for homogenising, easier-­to-­export linear narratives, as part of a wider synchronicity-­led initiative to dominate the global film market, often to the detriment of incipient narrative conventions emerging elsewhere in the world, that were soon replaced by their imported American counterparts (see McDonald 2006; Chung 2007 for accounts of how this acculturation process played out in Japan and Korea, respectively). Iranian film and cultural studies scholar Hamid Naficy (2003) has also examined the dynamics of film narrative colonisation and revealed the geopolitical import of this industrial strategy. In Naficy’s words: [A]t the heart of the US policy of technological transfer and development aid for the Third World since the 1950s, was this notion of homogenisation

From the “cinema of attractions” to danmu  99

and synchronicity of the world within Western consumerist ideology. This is a shift from the earlier policy of diachronicity, promoted by colonists, which tended to keep the developed and the under-­developed worlds apart. The emerging form of post-­ industrial capitalism sought synchronicity in the interest of creating global markets. (2003, 193) Hollywood’s imposition of the dominant narrational regime of Western modernity as a means to exercise maximum control over the reception and commercial performance of films required the enforcement of hierarchical power structures between authorship and viewership, and the technologies available at the time played an instrumental role in facilitating the expansion of this hierarchical regime of signification. The mutually influencing relationship between the technological and the social is aptly articulated by Stephen Heath, who notes that “[c]inema does not exist in the technological and then become this or that practice in the social; its history is a history of the technological and social together . . . in which the ideological is there from the start” (1981, 6). Unsurprisingly, the period during which the blueprint of American film narratives was drawn up featured the rise and consolidation of “recording technologies,” one of the three historically successive types of production technologies identified by Kress and van Leeuwen. Aimed primarily at the eye and the ear, recording technologies “allow more or less automated analogical representation” of what was presented as unmediated reality (Kress and van Leeuwen 2006, 217), thus providing the ideal inscription tools to enable the shift from diachronicity to synchronicity that underpins the narrational regime. Recording technologies placing synchrony at the centre of the temporal/ spatial order favour homogenising narratives and uphold the authority of the creator within existing media power structures. Acting as a form of “suture” (Heath 1981), they foster an ontology of representation restricting the range of relations that a film is constructed to establish with its viewers, ultimately endorsing an aesthetic of objectivity based on referential expressions of reality (Silverman 1988). Indeed, the emphasis of cinematography on synchronous diegetic sound as a way to keep the machinery and production process behind filmmaking hidden is typical of the “ontologies of referentiality, a view of representation being founded on direct, referential relations between the representations and the world” (Kress and van Leeuwen 2006, 218). In order to facilitate the implementation of this industrial/commercial strategy, the centrality of synchrony had to be extended to the subtitling of US film commodities. Minh-­ha coins the term “suture subtitling” to foreground the impact that the generalisation of the narrational film regime had on subtitling and the processes of multimodal meaning transfer that it entails (Minh-­ha 1992). The interplay between power structures and recording technologies during the formative years of subtitling allowed the film industry to “collapse, in subtitling, the activities of reading, hearing, and seeing into one single activity, as if they were all the same,” and hence “to naturalize a dominant, hierarchically unified

100 

Luis Pérez-­González

worldview” (Minh-­ha 1992, quoted in Nornes 1999, 18). Under the aesthetic of objectivity associated with recording technologies, subtitles have contributed to upholding the ontology of referentiality, acting as gatekeepers between the diegetic and the extra-­diegetic. The role of subtitles in the era of recording technologies has been conventionally confined to delivering in the target language approximate linguistic representations of the verbal meanings and/or communicative intentions encoded in the source text. The emphasis has been thus placed on the original message that filmmakers intend to convey through their characters’ speech – the assumption being that translators should privilege the “primordial” meaning expressed in the film dialogue. Further, while the contribution of visual and acoustic semiotic resources to the overall meaning of audiovisual texts is ostensibly acknowledged by audiovisual translation scholars (Taylor 2003; Pérez-­González 2007, 2014a, 2014b, 2020; Desilla 2012), professional subtitlers working for the film industry during the era of recording technologies have seen the scope of their mediation, and hence the exercise of their professional discretion, reduced to one single form of semiotic transfer – involving the re-­encoding of speech through written language.

3. Subtitling during the period of ontological transition Although the hegemony of the narrational regime continues to place stringent constraints on the multimodal configuration of commercial films, a growing body of documentary and ethnographic cinematic practices is revealing the extent to which “the sound track, like the image track, involves mediations, translations, and representational practices that push the sound into the realm of ideology” (Naficy 2004, 141), thus framing suturing synchrony as a commercial choice, rather than the only option available under the medial constraints of film. The formal organisation of Trinh Minh-­ha’s Surname Viêt Given Name Nam (1989), for example, “question[s] the representational nature of film” by breaking down mainstream filmmaking techniques and pointing to their flaws, therefore preventing the audience from suspending their disbelief (Vietnamese Cinema Blog 2015). Just as Trinh’s filming style tampers with the norms of cinematic realism (for example, by using lighting to disrupt the mood of a scene, camera movements that wander away from the subject and sound overlapping techniques), it also draws on superimposed titles and subtitles extensively, graphically and critically. Their large numbers and varied contents and layout give this film a truly calligraphic accent. Throughout, subtitles consisting of the translation of the film’s dialogue and voice-­over and of Vietnamese poetry and proverbs are displayed, as is customary, in the lower third of the screen. However, on many occasions, what the diegetic women say in Vietnamese or in heavily accented English is superimposed in different layouts, as blocks of English text on various regions of the film frame, including over the characters’ faces. These graphic titles, or what Trinh calls “visualized speech,” act as traditional subtitles by aiding

From the “cinema of attractions” to danmu  101

spectator comprehension. . . [although] they also serve other graphic, critical, and deconstructive functions. (Naficy 2004, 145–46) The ontology of referentiality is also critiqued in “accented films” shot by members of exilic and diasporic communities since the 1960s (Naficy 2004). Drawing on artisanal production techniques, accented creators adopt a performative stance involving subjective film shots (Naficy 2004, 136) and other techniques that enact a free indirect style in their films. Significantly, the deployment of counter-­hegemonic filmic discourses based on sound misalignment and de-­centred compositional patterns – that allows accented filmmakers to “comment upon cinema and reality, instead of just recording, reporting on, or representing reality” (146) – calls for the use of profuse on-­screen titles to channel “the problematic of linguistic, cinematic, and exilic translation and displacement” (146). While the erosion of the boundary between the diegetic and the extra-­diegetic can be accelerated in documentary and accented films by displaying subtitles deliberately out of synchrony, among other multimodal strategies, commercial films may also seek to undermine the narrational regime by exploring new forms of interplay between the story and the audience. Pablo Romero-­Fresco, for example, has explored new subtitling approaches where, in contrast to what happens in conventional films and drama series, the material dimension and the function of on-­screen titles are conceptualised prior to the production of the multimodal text they will be embedded in. Creative subtitling, as defined by Romero Fresco, adopts and extends certain features of ethnographic subtitling (MacDougall 1998, Chapter 7), primarily its capacity to foster new narrative mechanisms through which the authorial voice bypasses the diegetic characters to engage directly with the audience (Romero-­Fresco 2018). Creative subtitles – variously referred to in the literature as “decotitles” (Kofoed 2011), “authorial titles” (Pérez-­González 2012), “impact captions” (Sasamoto 2014; O’Hagan and Sasamoto 2016), or “integrated titles” (Fox 2016) – do not seek to deliver translations of the original spoken dialogue but rather to enhance aspects of characterisation or progress the narrative in more involving or immersive ways (Dywer 2015). Peter Kosminsky’s four-­part drama The State (2017) – which dramatises the experiences of four British men and women who leave their lives behind to join Islamic State in 2015 – illustrates the innovative use of authorial titles in mainstream media content. As the new recruits arrive in Syria and settle into the organisation, men begin combat training and the women are introduced to the society’s strict rules under the guidance of commanders and house leaders, respectively. English-­speaking characters going about their tasks make use of Arabic sentences pertaining to various rituals and cultural conventions associated with Islam, as a way to signal and reinforce their shared collective identity and community affiliation. When such expressions are used, stylised titles – that look “too aesthetic to be merely informative” (Crawley 2017) – display the transliterated Arabic sentences and their English translation in varying areas of the frame.

102 

Luis Pérez-­González

For example, when Shakira Boothe – a British doctor and single mother who joins the Islamic State in the hope of working in a state hospital – makes arrangements to see her first patients, she is told she cannot work without a mahram, at which point the title “mahram | male guardian” appears on screen. Another example of commercial film subverting the hierarchical power structures between the narrative and on-­screen titles is Paolo Sorrentino’s Il Divo (2008), a biographical drama about Giulio Andreotti, one of Italy’s most controversial Italian politicians, who found himself at the centre of a conspiratorial shady network that eventually led to his prosecution. Although the film sets out to re-­create the reality of key historical events, various semiotic resources are deployed from the performative perspective of auteurship. This is particularly striking in the opening sequence, that flashes a series of murder sequences in quick succession, complete with stylised captions providing details of the victims and the circumstances surrounding their death. Donald Clarke, for example, draws attention to the “artful captions that float behind buildings and emerge from car doors – offering indigestibly comprehensive descriptions of their [the victims’] role in the debasement of Italian politics” (Clarke 2009), while Mark Jenkins notes the resemblance between the playfulness of Sorrentino’s captions and those used in Danny Boyle’s Slumdog Millionaire (2008) (Jenkins 2009). As Millicent Marcus elaborates: [t]he pretense to investigative seriousness is immediately undercut by the playfulness of Sorrentino’s captions – the upside-­down hanging corpse of Roberto Calvi [one of the murder victims] is accompanied by a corresponddown caption, which turns right-­ sideup as the photogram ingly upside-­ rotates 180 degrees to its proper position. Similarly, the caption identifying Michele Sindona, poisoned in his jail cell, is shown backwards until the camera circles around the dying man so that we can read his name from left to right. Throughout the film, captions dance about the screen in a kind of freefloating semiotic abandon. Words, like images, have come untethered from their “signifieds.” By transforming the convention of the caption – hitherto understood as an indicator of its film’s unproblematic and unequivocal referentiality – into one more sign to be manipulated at will, Sorrentino asserts his ambiguous relationship to the truth claims of the documentary genre, and its offshoot in the films of cinema politico. (Marcus 2010, 253; emphasis in the original) The authorial titles deployed in The State and Il Divo enable the transgression of traditional narrative boundaries through “narrative metalepsis,” as conceptualised by Genette (1983 [1980]).1 Once they are incorporated into the mise en scène, authorial titles become an additional narrative plane intersecting with the unfolding diegetic events, dramatising the blurring of the boundary between fiction and reality. The creators of The State and Il Divo use authorial titles to transgress traditional narrative conventions and exploit the “double-­layeredness” of filmic communication (Vanoye 1985) to intrude in the diegetic action. The vertical level of interpersonal

From the “cinema of attractions” to danmu  103

communication (between the filmmakers and the audience) is prioritised in places over its horizontal counterpart (between the diegetic characters), as titles provide viewers with information and knowledge about the fictional world that would normally be confined to the diegetic characters. The combined use of cinematography and metaleptic captions in Il Divo, for instance, disrupts classical hierarchies in film narratives, undermining the ontological status of fictional subjects. As Ambrose Hogan argues, criminal allegations are not made against specific individuals through diegetic speech (Hogan 2009). Instead, they are communicated to the viewer through the interplay between visual signifying means and performative titles, some of which “are spat out of the mouth of a victim of poisoning, or swing from under Blackfriars Bridge along with the cooling corpse of [Italian banker] Roberto Calvi” (Hogan 2009). The previous examples have shown how the deployment of titles can be used to destabilise the ontology of referentiality for various purposes, even in the context of media content produced for commercial release. Rather than simply contributing to the representation of a linear narrative or “hiding behind the pretense of an unacknowledged spectator” (Gunning 1993, 5), authorial titles comment upon the diegetic and extra-­diegetic realities, promoting the same “parataxic” reading experience as the primitive intertitles featuring in films of the cinema of attractions (Grillo and Kawin 1981). Whether it is through the delivery of diegetic subtitles out of synchrony or in unusual regions of the frame (as is the case with Trinh Minh-­ha’s “visualised speech” [Naficy 2001, 123]), or through the deployment of authorial titles that challenge the privileged status of the diegetic narrative over its viewers (creating new spectatorial experiences where creators interpellate the audience directly), films and serialised dramas are presenting audiences with new processing demands. Instead of coherent single gestalts, viewers are faced with multimodal artefacts where the strands of meaning conveyed through the juxtaposition (rather than integration) of different signifying means must be dissected and interpreted independently – thus heightening the potential for subjective interpretations of the diegetic narrative.

4. Subtitling in the era of synthesising technologies The advent of digitisation has brought about the emancipation of subtitling from the media industries. In the digital culture, media content is increasingly shaped by variable degrees of convergence between industrial and amateur practices, with growing numbers of ordinary people becoming involved in its appropriation, annotation (including subtitling), editing, and distribution (Pérez-­González 2014a, 2017; Dwyer 2017a). Driven by the ubiquity of relevant technological tools, amateur (often participatory) subtitling has quickly reconfigured individual viewership as part of wider processes of social interaction around the production and consumption of media content, turning the “spectatorial culture into participatory culture” (Jenkins 2006, 41). Under this new postmodernist aesthetic, where subtitled media content is often watched via social media platforms, “net

104 

Luis Pérez-­González

users [including amateur subtitlers] are increasingly disinterested in the metanarrative (content) and instead become obsessed with multiple small narratives (comments)” (Xu 2016, 439). In this context, media content represents an opportunity for the negotiation of intersubjectivity – understood as a shared and reciprocal perception of the meaning created through joint activity and socially managed interaction – among members of virtual, geographically dispersed communities of interest. In selecting what is to be subtitled and steering decision-­making processes on the materiality and wording of their subtitles, these agencies subordinate the authority of the original to the subtitles’ capacity to mobilise political or playful affinity through the online comment facility that accompanies the video. Participation, in short, takes over spectatorship. Amateur subtitling is typically executed by virtual communities that capitalise on the affordances of networked communication to exploit their members’ skills sets or collective intelligence. Subtitlers involved in this form of “co-­creative labour” (Banks and Deuze 2009) draw on their linguistic proficiency – which does not necessarily equate with translation competence – and activist orientation. “Aesthetic activism” (Pérez-­González 2014a), one of the two main strands covered in the literature, is best illustrated by “fansubbing,” a term traditionally associated with the prolific global subculture built around the subtitling of Japanese anime by fans of this genre and, in more recent times, with a form of immaterial subtitling labour undertaken by followers of commercial dramas, series, or musical genres (Barra 2009). Fansubbing practices are normally fuelled by a desire to tamper with commercially translated media content that fans-­turned-­translators regard as “culturally odourless” (Iwabuchi 2002, 27). Fansubbing practices have been shown to experiment with the formal dimension of subtitles (Dwyer 2012) and bestow a high degree of visibility on translators, who maximise the deconstructive affordances of digital technologies to act as intercultural brokers between the text producers and users from inside the frame. On the other hand, the archival, annotation (through subtitles), and recirculation of media content are also central to the activities of networks of politically engaged amateur subtitlers, for digital culture has facilitated the rise and consolidation of networks of activist subtitlers seeking “to elaborate and practise a moral order in tune with their own narratives of the world” (Baker 2006, 481). Pérez-­González (2010, 2013, 2016) has shown how participatory networks of activist subtitling resist the dynamics of the news media industry and challenge the control that global corporations have traditionally exerted over the distribution and consumption of the content they produce. These participatory communities take on the role of self-­appointed translation commissioners and selectively appropriate the content they wish to subtitle and redistribute among strategically targeted linguistic constituencies that otherwise would not have been able to access that content through mainstream or commercial media circuits. In the same vein, Mona Baker has explored how subtitlers embedded in various activist collectives contribute to exploring prefigurative principles (including horizontality, solidarity, and rejection of representational modes of practice), choosing to bring their mediation

From the “cinema of attractions” to danmu  105

into light, “rather than obscure it and lull the viewer into the illusion of a direct, unmediated experience” of the subtitled content (Baker 2018, 460–61). As was also the case with aesthetic activists, politically engaged subtitlers ultimately work with their viewers towards the co-­construction of affinity spaces for the negotiation of their individual and collective identities. Unconstrained by commercial and financial interests, amateur subtitling by ordinary citizens is contributing to foreground a postmodern aesthetic based on “the primacy of our mundane day-­to-­day experiences over that of the rigid structures” favoured by the media industry and the narratives that embody their cultural and economic hegemony (Xu 2016, 439). This change in the role that subtitling plays within the multimodal configuration of audiovisual texts is consistent with other manifestations of “citizen media” practices, which encompass [t]he physical artefacts, digital content, practices, performative interventions and discursive formations of affective sociality produced by unaffiliated citizens as they act in public space(s) to effect aesthetic or socio-­political change or express personal desires and aspirations, without the involvement of a third party or benefactor. It also comprises the sets of values and agendas that influence and drive the practices and discourses through which individuals and collectivities position themselves within and in relation to society and participate in the creation of diverse publics. (Baker and Blaagaard 2016, 16) As noted earlier, citizen subtitlers – a term adopted here to encompass ordinary citizens-­turned-­subtitlers working with different agendas – subscribe to a postmodern aesthetic of mundane playfulness or political engagement that often ignores the constraints of narrative motivation and the concomitant need for synchronous temporal and spatial relationships between the original speech and its subtitled version. Indeed, the status of the original is subverted by the comment culture that amateur subtitling practices seek to foster and stimulate. Positing spectatorial subjectivity through parataxic reading practices is at odds with the ontology of referentiality underpinning the production of classical films and therefore requires the use of “synthesizing technologies” (Kress and van Leeuwen 2006, 217). Against the crisis of representation that the shift from recording to synthesising technologies has prompted, digitisation enables alternative forms of “representation-­as-­design” (Kress and van Leeuwen 2006, 217) that open up new possibilities for more democratic spectatorial engagement by bringing the world of the story closer to the space of the audience. The ontology of referentiality associated with the narrational regime is thus superseded by an ontology of deconstruction under which ordinary citizens can watch and produce subtitles, and the diegetic world is no longer closed off to viewers, who can make use of digital technologies to experiment with and expose the multimodal circuitry of the multimodal text, “deconstructing the combinatorial possibilities and laying bare their cultural/social sources” (Kress and van Leeuwen 2006, 219).

106 

Luis Pérez-­González

5. The multimodal implications of design and rhetoric in Chinese danmu In digital media habitats, where amateur rhetors have gained greater visibility and influence, subtitling is emerging as a meaning-­making practice that contributes to the deconstruction of the original text by exposing the interstices around its multimodal wiring. As reported in the literature surveyed in the previous section, the ever-­closer alignment between amateur subtitling and other manifestations of citizen media entails the adoption of experimental practices that mobilise a range of signifying means to style on-­screen text in unconventional ways for maximum effect. In terms of the ontological value that they attribute to the source text, citizen subtitlers steer us away from the individual or subject position of their professional counterparts, towards collective discursive spaces of translatorship involving complex negotiations of mundane or ethical identity. Technological developments – whether in the form of proprietary web-­based captioning tools such as Amara (Amara n.d.), that have been made available to support volunteer initiatives as a way of accruing symbolic capital in the community; freeware editing and subtitling applications that citizen subtitling groups have chosen to integrate within their collective workflows; or social media platforms providing the tools to develop a participatory comment culture around the consumption of subtitled output – have been instrumental in delivering this ontological shift. By enabling users to “select from a range of different semiotic resources” as well as “incorporat[ing] and represent[ing] knowledge about what constitutes effective use of these resources” in citizen subtitling, these tools qualify as “semiotic software” (Djonov and van Leeuwen 2017, 567). As befits synthesising technologies, semiotic software has its own semiotic regime, insofar as the “rules for using the semiotic resources available within a software product are externalised in and tacitly imposed by its design” (Djonov and van Leeuwen 2017, 571). Significantly, semiotic software is not only important for enabling multimodal affordances and setting restrictions in terms of how various resources – including, but not limited to, colour, font, texture, or animation – can be jointly deployed (van Leeuwen, Djonov, and O’Halloran 2013), but also for being semiotic artefacts in their own right with the capacity to influence social practices (Poulsen, Kvåle, and van Leeuwen 2018, 596). The dialectic between the affordances and constraints imposed by semiotic software and the social practices that it enables will be explored in the remainder of this chapter through the lens of Kress and van Leeuwen’s (2006) well-­established analytical notions of “design” and “rhetoric.” In the context of this study, the former designates the multimodal meaning-­making potential of the technological resources facilitating the work of ordinary people involved in participatory subtitling, whereas the latter will be used to foreground the scope and complexity of the social relations that obtain around the production and consumption of subtitled media content. As the practices associated with successive fansubbing turns (Dwyer 2018) and political subtitling (Pérez-­González 2010, 2013, 2016; Baker 2018) are relatively

From the “cinema of attractions” to danmu  107

well documented in the literature, the remainder of this paper examines the interplay between semiotic technology (design) and social practices (rhetoric) – and the ontology of deconstruction that they bring about – as instantiated in the context of the under-­explored Chinese participatory subculture known as danmu. Craig Howard describes danmu – also known as “barrage commenting” (Li 2017; Dwyer 2017b) – as a video annotation system that was first launched in 2007 by a Japanese ACG (anime, comic, game) video-­sharing website (Howard 2012). The system, which was introduced in China in 2010 by digital platforms like Bilibili.com, involves the display of viewer-­generated titles on the very screen where the media content in question is being played. The titles – which, as elaborated further, can deliver translations of the diegetic dialogue or, alternatively, viewers’ comments on a range of issues more or less directly connected with the video they are inscribed on – are therefore superimposed on the multimodal text, rather than in a separate section, as is the case in YouTube. According to Jannis Androutsopoulos, danmu – a term that designates both the platform allowing for the generation and sharing of comments, as well as the comments themselves – constitutes a form of “participatory spectacle” where media content is reduced to a simple background, with the production and reading of comments taking over as the focal attraction (Androutsopoulos 2013). Indeed, the climatic parts of a series episode or film may bring about so many danmu that comments may form a multi-­layered “bullet curtain” (which is how danmaku, the original term used in Japanese to refer to this practice, translates into English) that may become thick enough to block the visuals (see Figure 4.1).

FIGURE 4.1 

Screenshot of video material featuring danmu

Source: (Bilibili.com)

108 

Luis Pérez-­González

In what follows, the analysis of the danmu aesthetic as a form of representation-­ as-­design will be informed primarily by Stöckl’s (2004) typology of multimodal resources – a framework that, whilst not primarily developed to analyse audiovisual texts, provides a productive and systematic set of analytical tools based on Stöckl’s conceptualisation of semiotic resources as constituents of hierarchical networks of choices. As is also the case with mainstream and citizen subtitling, the danmu titling culture mobilises a range of modes, i.e., “sign systems from which communicators can pick their signs to realise their communicative intentions” (Stöckl 2004, 11). Language and image remain as core modes – understood as those types of signifying means that are “deeply entrenched in people’s popular perceptions of codes and communication” (Stöckl 2004, 14) – in the danmu environment, although the prominence of the user-­generated titles signals very clearly (i) how traditional relationships between authorship and viewership are being subverted; and (ii) the extent to which “the inner layer (metanarrative) loses primacy to the surface layer (nonnarrative)” (Xu 2016, 448) that viewers generate. The fact that danmu titles seek primarily to negotiate a perceived sense of intersubjectivity among viewers, rather than to enjoy the media content per se, entails a clear asymmetry between the volume of information conveyed through each of the two sensory channels at play in this site, with the written word (i.e., the medial realisation of the language mode within the visual channel) taking priority over its auditory counterpart. The inscription of danmu on top of the audiovisual text foregrounds the crisis of the representational regime and, more specifically, the emancipation of titles from the primordial narrative. As is also the case in other forms of citizen subtitling, the Bilibili platform – i.e., the semiotic software driving the danmu culture – allows for a range of multimodal enhancements through the activation of one or more title submodes, namely “the building blocks of a mode’s grammar” (Stöckl 2004, 14). Font and colour choices, for example, are among the most productive submodes that viewers can use to style written text and to construct a visual identity for themselves amid danmu that have been previously posted by other viewers. Interestingly, in this environment characterised by the proliferation of superimposed comments flowing horizontally over the video, from the right to the left of the screen, it is static written text that becomes relatively salient and commands a higher degree of ontological authority. Static danmu – which can only be mobilised by Bilibili premium users – feature at the top and bottom of the frame (Wu et al. 2018) to deliver explanations on both general and specific aspects of the video (Ma and Cao 2017) and to deliver translations of the diegetic speech by way of traditional subtitles (Wu et al. 2018), respectively. Capitalising on the affordances enabled by semiotic software, members of the danmu subculture act as rhetors making cumulative decisions pertaining to the deployment of semiotic resources among the network of submodal options available at any given point. The danmu conglomerate therefore emerges as a composite artefact moulded by the decisions made by users during the design stage – as is also the case with other forms of citizen subtitling. Ma and Cao, for example, report

From the “cinema of attractions” to danmu  109

that users playing specific roles (e.g., uploader) can stick to the same colour “when commenting repeatedly throughout the video”; groups of posters involved in a shared task, such as translating the lyrics of a diegetic song into different languages and dialects through danmu may choose to exhibit a shared visual identity through the adoption of the same colour-­cum-­font pattern; and individual users wishing to interact as part of a self-­contained group can signal their shared membership by prefacing their danmu with strings of words (Ma and Cao 2017, 779). In these cases, then, the selection of fonts and colours may seek to reflect what is being talked about, or the relationship between the interactants, just as in other forms of citizen subtitling. But the deconstructing role of synthesising technologies in the danmu platforms also has some liberating effects: submodal choices are not bound by the need to ensure optimum visibility and readability conditions, as in chaotic barrage commenting environments “the consumption of the metanarrative [the diegetic plot] is cast away in favour of the irrelevant and carnivalesque aspect” of users’ danmu (Xu 2016, 439). From a temporal perspective, there is a difference between static, subtitle-­like danmu featuring at the bottom of the frame – which are delivered in synchrony with diegetic speech – and non-­translational danmu occupying the entire screen and often blocking out the actual video. Users wishing to post danmu do so by linking the display time for their contribution to the specific time code of the video moment that it comments on. New danmu can be added all the time, but previously logged ones will be replayed every time the video is watched, thus turning the interface into an ever-­g rowing archive of user-­generated contributions. As Jinying Li explains, “the temporality structured by the danmaku interface leads to a collective user experience of ‘virtual liveness’ ”: while all past danmu are shown simultaneously, even though there might be important gaps between the times in which they were inscribed, it is impossible for users to engage in live conversations. Danmu posted in response to a previous comment may never be read by the latter’s author, so interacting in this environment effectively amounts to debating “with someone who spoke in the past but whose words always appear in the present” (Li 2017, 248). In overstepping the boundaries between the diegetic and the extra-­diegetic, danmu exhibits many similarities to other forms of title-­based mediation driven by a deconstructive ontology examined in previous sections, including authorial titles, understood as a transitional regime between recording and synthesising technologies, and citizen subtitling, as performed by communities galvanised around both fandom or political affinity. But while the network of social relations that emerge among participants in citizen subtitling has been theorised in some depth (Pérez-­ González 2014a), the rhetorical dimension of danmu has not yet been explored. In the remainder of this section, the scope of the discussion on the rhetorical construction of danmu will be limited to a specific dimension, namely, the politics of “mutual recognition” (Thrift 2009). Amateur subtitling communities have been found to resemble other participatory networks that operate in the citizen media culture insofar as they index

110 

Luis Pérez-­González

“self-­referential properties” to their preferred “values, beliefs and practices” (Deuze 2006, 71). Enhancing the visibility of amateur prosumers, in their capacity as representatives of a collective identity shared by subtitlers and their audiences, is therefore crucial to the proliferation of participatory subtitling. Essentially, the drive to enforce an aesthetic or a politics of mutual recognition between subtitlers and viewers empowers amateur translator networks to adopt an interventionist mediation approach, moulding media experiences to fit viewers’ expectations. By drawing on the affordances of synthesising technologies and reflexively engaging in the manipulation of media content that circulates in their environment, ordinary people are able to establish new participatory sites for the expression of subjective spectatorial experiences and promote, as is also the case with other citizens engaged in forms of self-­mediation that do not involve translation, “new forms of playful citizenship, critical discourse and cosmopolitan solidarity” (Chouliaraki 2010, 227). Danmu take the politics of recognition much further, as illustrated by the abundant textual traces of viewer investment in and engagement with the video playing in the background. Viewers have been shown to rely on danmu for different purposes (Wu et al. 2018). In some cases, they post comments serving the same function as traditional subtitles, i.e., delivering Chinese versions of the original diegetic speech – that, as noted earlier, will often end up featuring near the bottom of the frame. The collaborative nature of the subtitling process undertaken through danmu has been documented by Wu et al. (2018), who report on groups of users collaboratively creating Chinese “subtitles” for different sections of the same video by inscribing their danmu on the relevant fragments. Plotters, however, use danmu to issue warnings about imminent plot twists, typically by framing such climatic events in terms of the emotional reactions they are likely to elicit. Examples include: “ ‘High-­energy reaction ahead!!!’ (i.e., the following plot is very exciting), ‘High-­energy reaction ahead, please wear a helmet!!!’ (i.e., the following plot is very exciting and intense, please be prepared mentally), [and] ‘Tut tut, see the end from here’ (i.e., the end can be guessed through this plot)” (Wu et al. 2018, 212). Enforcing community regulation norms and participating in “parasocial interaction” – where users engage in conversation with imaginary interlocutors or with diegetic characters – are other purposes that danmu may serve. The great bulk of danmu inscribed on any video, however, contributes to the materialisation of a site that enacts the aesthetic of mutual recognition, where viewers become visible to each other when attempting to learn foreign languages, sharing jokes or criticisms, and other more or less mundane experiences. As a textually and formally disruptive contrivance moulded by the interplay between design and rhetoric, danmu environments contribute to the formation of spaces of expressivity and interventionist mediation enabled by the co-­existence of diegetic (translational) titles and extra-­diegetic posts. In their search for mutual recognition, viewers of media content in the digital culture build communities of rhetors by exploring the affordances of the semiotic software and gauging the potential and limitations of barrage commenting.

From the “cinema of attractions” to danmu  111

6. Conclusion As the ontology of referentiality is replaced by one of deconstruction, synthesising technologies are exposing the crisis of the “representational” role that subtitles have played within traditional audiovisual ensembles. Although experimental subtitling practices can also be observed in mainstream media content, the proliferation of citizen subtitling practices and participatory environments like danmu is foregrounding interventionist reconfigurations of multimodal textualities driven by aesthetic and/or political allegiances. Further, the dynamics of subtitling in the digital culture are also drawing attention to the extent to which semiotic software – a semiotic artefact in its own right – is influencing the practices through which viewers choose to engage with texts and with fellow viewers. The popularity of danmu platforms raises important questions for subtitling studies scholars regarding the interplay between danmu that serve the same function as a translational subtitle and other more parasocial comments. Barrage titling, and the obscuring of the multimodal text that they are inscribed on, problematises the long-­standing contention among industry players that subtitles represent a form of intrusion in the audiovisual ensemble. Debates concerning the need for commercial subtitles to prioritise clarity and legibility – not least by jettisoning verbal references to acoustic and visual signifying resources that mainstream audiences can access directly – should gauge whether, and to what extent, viewers’ capacity to establish intermodal connections is being affected. Even if we accept that danmu are ushering in “a form of postmodern aesthetic that favours eclectic, user-­driven experiences that are detached from the actual meaning or narrative of the products they are actually consuming” (Xu 2016, 447), (i) understanding the specific impact of calligraphic overflow and spectatorial saturation, and (ii) examining how a potential contagion of the barrage culture outside of China might affect the production of the primordial audiovisual narratives are bound to emerge as important disciplinary concerns. The development of these new textualities which, as noted previously, enable the collaborative production and critique of subtitles may open new avenues for the expansions of translation crowdsourcing models that have not been exploited to date. Equally important are the methodological challenges involved in the study of community formation by tracing and connecting multiple overlaid layers of written text whose authors and identities are not yet retrievable. Beyond these specific issues, the emergence and popularisation of danmu, so far restricted to China, will provide translation scholars who are interested in the internationalisation of disciplinary discourses with an opportunity to theorise this new regime of signification based on an “aesthetic of hyperflatness” (Xu 2016, 447) and informed by East Asian artistic traditions that works not only towards “the direct subversion of the metanarrative but the very nascency of the Chinese postmodern consciousness” (449).

112 

Luis Pérez-­González

Note 1 See O’Sullivan (2011) and Pérez-­González (2014a) for a more extended analysis of how subtitling enables this form of metalepsis.

References Amara. n.d. Amara Subtitling Software. https://amara.org/en/. Androutsopoulos, Jannis. 2013. “Participatory Culture and Metalinguistic Discourse: Performing and Negotiating German Dialects on Youtube.” In DISCOURSE 2.0 Language and New Media, edited by Deborah Tannen and Anna Marie Trester, 47–73. Washington, DC: Georgetown University Press. Baker, Mona. 2006. Translation and Conflict: A Narrative Account. London and New York: Routledge. ———. 2018. “Audiovisual Translation and Activism.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-­ González, 453–67. London and New York: Routledge. Baker, Mona, and Bolette Blaagaard. 2016.“Reconceptualizing Citizen Media: A Preliminary Charting of a Complex Domain.” In Citizen Media and Public Spaces: Diverse Expressions of Citizenship and Dissent, edited by Mona Baker and Bolette Blaagaard, 1–22. London and New York: Routledge. Banks, John, and Mark Deuze. 2009. “Co-­creative Labour.” International Journal of Cultural Studies 12 (5): 419–31. Barra, Luca. 2009. “The Mediation Is the Message: Italian Regionalization of US TV Series as Cocreational Work.” International Journal of Cultural Studies 12 (5): 509–25. Chouliaraki, Lilie. 2010. “Self-­mediation: New Media and Citizenship.” Critical Discourse Studies 7 (4): 227–32. Chung, Sung-­ill. 2007. “Four Variations on Korean Genre Film: Tears, Screams,Violence and Laughter.” In Korean Cinema from Origins to Renaissance, edited by Mee-­hyun Kim, 1–14. Seoul: Communication Books. Clarke, Donald. 2009. “Il Divo,” Irish Times, March 20. www.irishtimes.com/culture/ il-­divo-­1.726848. Crawley, Peter. 2017. “The State Review: What Has Driven 1,000 UK People to Fight with Isis?” The Irish Times, August  24. www.irishtimes.com/culture/tv-­radio-­web/ the-­state-­review-­what-­has-­driven-­1-­000-­uk-­people-­to-­fight-­with-­isis-­1.3196993. Cronin, Michael. 2003. Translation and Globalization. London: Routledge. Desilla, Louisa. 2012.“Implicatures in Film: Construal and Functions in Bridget Jones Romantic Comedies.” Journal of Pragmatics 44: 30–53. Deuze, Mark. 2006. “Participation, Remediation, Bricolage: Considering Principal Components of a Digital Culture.” The Information Society 22: 63–75. Djonov, Emilia, and Theo van Leeuwen. 2017. “The Power of Semiotic Software: A Critical Multimodal Perspective.” In The Routledge Handbook of Critical Discourse Analysis, edited by John Flowerdew and John E. Richardson, 566–81. London and New York: Routledge. Dwyer, Tessa. 2005. “Universally Speaking: Lost in Translation and Polyglot Cinema.” Linguistica Antverpiensia 4: 295–310. ———. 2012. “Fansub Dreaming on Viki: ‘Don’t Just Watch but Help When You Are Free’.” In “Non-­professional Translation,” special issue edited by Şebnem Susam-­Saraeva and Luis Pérez-­González. The Translator 18 (2): 217–43. ———. 2015. “From Subtitles to SMS: Eye Tracking, Texting and Sherlock.” Refractory: A Journal of Entertainment Media 25. Accessed June 20, 2018. http://refractory.unimelb. edu.au/2015/02/07/dwyer/.

From the “cinema of attractions” to danmu  113

———. 2017a. Speaking in Subtitles: Revaluing Screen Translation. Edinburgh: Edinburgh University Press. ———. 2017b. “Hecklevision, Barrage Cinema and Bullet Screens: An Intercultural Analysis.” Participations: Journal of Audience and Reception Studies 14 (2): 571–89. www.participations. org/Volume%2014/Issue%202/29.pdf. ———. 2018. “Audiovisual Translation and Fandom.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-­González, 436–52. London and New York: Routledge. d’Ydewalle, Géry and Wim De Bruycker. 2007. “Eye Movements of Children and Adults While Reading Television Subtitles.” European Psychologist 12: 196–205. Fox,Wendy. 2016. “Integrated Titles – An Improved Viewing Experience? A Contrastive Eye Tracking Study on Traditional Subtitles and Integrated Titles for Pablo Romero-­Fresco’s Joining the Dots.” In Eyetracking and Applied Linguistics, Open Access Book Series Translation and Natural Language Processing, edited by Silvia Hansen-­Schirra and Sambor Grucza, 5–30. Berlin: Language Science Press. Gambier,Yves. 2006.“Multimodality and Subtitling.” In Proceedings of the Marie Curie Euroconferences MuTra: Audiovisual Translation Scenarios, edited by Mary Carroll, Heidrun Gerzymisch-­ Arbogast and Sandra Nauert. Copenhagen May 1–5. Accessed September 15, 2013. www. euroconferences.info/proceedings/2006_Proceedings/2006_proceedings.htm. Gambier, Yves, and Sara Ramos Pinto. 2016. “Introduction.” Target: International Journal in Translation Studies 28 (2): 185–91. Genette, Gérard E. 1983 [1980]. Narrative Discourse: An Essay in Method. Translated by Jane E. Lewin. Ithaca, NY: Cornell University Press. Grillo, Virgil, and Bruce Kawin. 1981. “Reading at the Movies: Subtitles, Silence and the Structure of the Brain.” Post Script: Essays in Film and the Humanities 1 (1): 25–32. Grochowski, Thomas. 2007. “Horse Feathers, Monkey Business and Lion Taming: The Marxes in Transition.” In A Century of the Marx Brothers, edited by Joseph Mills, 53–71. Newcastle upon Tyne: Cambridge Scholars Publishing. Gumbrecht, Hans Ulrich. 2004. Production of Presence:What Meaning Cannot Convey. Stanford: Stanford University Press. Gunning, Tom. 1986. “The Cinema of Attraction: Early Film, its Spectator and the Avant-­ Garde.” Wide Angle 8 (3): 58–70. ———. 1993. “ ‘Now You See It, Now You Don’t’: The Temporality of the Cinema of Attractions.” The Velvet Light Trap 32: 4–12. Heath, Stephen. 1981. Questions of Cinema. Bloomington: Indiana University Press. Hogan, Ambrose. 2009. “Il Divo.” Thinking Faith, April 1. www.thinkingfaith.org/articles/ film_20090401_1.htm. Howard, Craig D. 2012. “Higher Order Thinking in Collaborative Video Annotations: Investigating Discourse Modeling and the Staggering of Participation.” Unpublished PhD Thesis, Indiana University Bloomington. https://eric.ed.gov/?id=ED552082. Iedema, Rick. 2003. “Multimodality, Resemiotization: Extending the Analysis of Discourse as Multi-­semiotic Practice.” Visual Communication 2 (1): 29–57. Iwabuchi, Koichi. 2002. Recentering Globalization: Popular Culture and Japanese Transnationalism. Durham, NC: Duke University Press. James, Meg. 2001. “Language Barrier Slows Movie Releases.” Los Angeles Times, August 25. Accessed June 20, 2018. http://articles.latimes.com/2001/aug/25/business/fi-­38050Y. Jenkins, Henry. 2006. Fans, Bloggers, and Gamers: Exploring Participatory Culture. New York: NYU Press. Jenkins, Mark. 2009. “Powerful Mystery Man of Italian Politics.” April 24. www. npr.org/templates/stor y/stor y.php?stor yId=103290407&t=155423705174 2&t=1554763303526.

114 

Luis Pérez-­González

Jensema, Carl, Sameh E. Sharkawy, Ramalinga Sarma Danturthi, Robert Burch, and David Hsu. 2000. “Eye Movement Patterns of Captioned Television Viewers.” American Annals of the Deaf 145 (3): 275–85. Kofoed, D. T. 2011. “Decotitles, the Animated Discourse of Fox’s Recent Anglophonic Internationalism.” Reconstruction 11 (1): n.p. Kress, Gunther, and Jon Ogborn. 1998. “Modes of Representation and Local Epistemologies: The Presentation of Science in Education.” In Subjectivity in the School Curriculum, Working Paper 2, Institute of Education. London: University of London. Kress, Gunther, and Theo van Leeuwen. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold. ———. 2006. Reading Images: The Grammar of Visual Design, 2nd ed. London and New York: Routledge. Kruger, Jan-­Louis. 2018. “Eye Tracking in Audiovisual Translation Research.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-­González, 350–66. London and New York: Routledge. Li, Jinying. 2017. “The Interface Affect of a Contact Zone: Danmaku on Video-­Streaming Platforms.” Asiascape: Digital Asia 4: 233–56. Littau, Karin. 2016a. “Translation and the Materialities of Communication.” Translation Studies 9 (1): 82–96. ———. 2016b. “Translation’s Histories and Digital Futures.” International Journal of Communication 10: 907–28. Ma, Xiaojuan, and Nan Cao. 2017. “Video-­based Evanescent, Anonymous, Asynchronous Social Interaction: Motivation and Adaption to Medium.” In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, 770–82. https:// dl.acm.org/citation.cfm?id=2998256. MacDougall, David. 1998. “Subtitling Ethnographic Films.” In Transcultural Cinema, Princeton, edited by David MacDougall, 165–78. Princeton, NJ: Princeton University Press. Marcus, Millicent. 2010. “The Ironist and the Auteur: Post-­Realism in Paolo Sorrentino’s Il Divo.” The Italianist 30 (2): 245–57. Mason, Ian. 2001. “Coherence in Subtitling: The Negotiation of Face.” In La traducción en los medios audiovisuales, edited by Frederic Chaume and Rosa Agost, 19–32. Castelló de la Plana: Servei de Publicacions de la Universitat Jaume I. McDonald, Keiko. 2006. Reading a Japanese Film: Cinema in Context. Honolulu: University of Hawai’i Press. Minh-­ha, Trinh T. 1992. Framer Framed. London and New York: Routledge. ———. 2005. The Digital Film Event. London and New York: Routledge. Mubenga, Kajingulu Somwe. 2009. “Towards a Multimodal Pragmatic Analysis of Film Discourse in Audiovisual Translation.” Meta 54 (3): 466–84. Musser, Charles. 2006. “Rethinking Early Cinema: Cinema of Attractions and Narrativity.” In The Cinema of Attractions: Reloaded, edited by Wanda Strauven, 389–416. Amsterdam: Amsterdam University Press. Naficy, Hamid. 2001. An Accented Cinema: Exilic and Diasporic Filmmaking. Princeton, NJ: Princeton University Press. ———. 2003. “Theorizing ‘Third World’ Film Spectatorship: The Case of Iran and Iranian Cinema.” In Rethinking Third Cinema, edited by Anthony R. Guneratne and Wimal Dissanayake, 183–201. London: Routledge. ———. 2004. “Epistolarity and Textuality in Accented Films.” In Subtitles: On the Foreignness of Film, edited by Atom Egoyan and Ian Balfour, 131–51. Cambridge, MA and London: The MIT Press.

From the “cinema of attractions” to danmu  115

Nornes, Abé Mark. 1999. “For an Abusive Subtitling.” Film Quarterly 52 (3): 17–34. O’Hagan, Minako, and Ryoko Sasamoto. 2016. “Crazy Japanese Subtitles? Shedding Light on the Impact of Impact Captions with a Focus on Research Methodology.” In Eyetracking and Applied Linguistics, Open Access Book Series Translation and Natural Language Processing, edited by Silvia Hansen-­Schirra and Sambor Grucza, 31–58. Berlin: Language Science Press. O’Sullivan, Carol. 2011. Translating Popular Film. London: Palgrave Macmillan. O’Sullivan, Carol, and Jean-­François Cornu. 2018. “History of Audiovisual Translation.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-­González, 15–30. London and New York: Routledge. Perego, Elisa, Fabio del Missier, Marco Porta, and Mauro Mosconi. 2010. “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology 13: 243–72. Pérez-­González, Luis. 2007. “Intervention in New Amateur Subtitling Cultures: A Multimodal Account.” Linguistica Antverpiensia 6: 67–80. ———. 2010. “Ad-­hocracies of Translation Activism in the Blogosphere: A Geneological Case Study.” In Text and Context: Essays on Translation and Interpreting in Honour of Ian Mason, edited by Mona Baker, Maeve Olohan and María Calzada Pérez, 259–87. Manchester: St. Jerome. ———. 2012. “Co-­Creational Subtitling in the Digital Media:Transformative and Authorial Practices.” International Journal of Cultural Studies 16 (1): 3–21. ———. 2013. “Amateur Subtitling as Immaterial Labour in Digital Media Culture: An Emerging Paradigm of Civic Engagement.” Convergence: The International Journal of Research into New Media Technologies 19 (2): 157–75. ———. 2014a. Audiovisual Translation: Theories, Methods, Issues. London and New York: Routledge. ———. 2014b. “Multimodality in Translation and Interpreting Studies: Theoretical and Methodological Perspectives.” In A Companion to Translation Studies, edited by Sandra Bermann and Catherine Porter, 119–31. Chichester: Wiley-­Blackwell. ———. 2016. “The Politics of Affect in Activist Amateur Subtitling: A Biopolitical Perspective.” In Citizen Media and Public Spaces: Diverse Expressions of Citizenship and Dissent, edited by Mona Baker and Bolette Blaagaard, 118–35. London and New York: Routledge. ———. 2017. “Investigating Digitally Born Subtitling Agencies in the Context of Popular Culture.” In Advances in Amateur Subtitling, edited by David Orrego Carmona and Yvonne Lee, 15–36. Newcastle: Cambridge Scholars. ———. 2020. “Multimodality.” In The Routledge Encyclopedia of Translation Studies, 3rd ed., edited by Mona Baker and Gabriela Saldanha, n.p. London and New York: Routledge. Poulsen, Søren Vigild, Gunhild Kvåle, and Theo van Leeuwen. 2018. “Introduction.” In “Social Media as Semiotic Technology,” special issue edited by Søren Vigild Poulsen, Gunhild Kvåle and Theo van Leeuwen. Social Semiotics 28 (5): 593–600. Remael, Aline. 2001. “Some Thoughts on the Study of Multimodal and Multimedia Translation.” In (Multi) Media Translation: Concepts, Practices and Research, edited by Yves Gambier and Henrik Gottlieb, 13–22. Amsterdam: John Benjamins. ———. 2003. “Mainstream Narrative Film Dialogue and Subtitling.” The Translator 9 (2): 225–47. Romero-­Fresco, Pablo. 2018. “Accessible Filmmaking: Translation and Accessibility from Production.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-­ González, 96–113. London and New York: Routledge. Sasamoto, Ryoko. 2014. “Impact Caption as a Highlighting Device: Attempts at Viewer Manipulation on TV.” Discourse, Context and Media 6: 1–10.

116 

Luis Pérez-­González

Silverman, Kaja. 1988. The Acoustic Mirror: The Female Voice in Psychoanalysis and Cinema. Bloomington: Indiana University Press. Stöckl, Hartmut. 2004. “In Between Modes: Language and Image in Printed Media.” In Perspectives on Multimodality, edited by Eija Ventola, Cassily Charles and Martin Kaltenbacher, 9–30. Amsterdam and Philadelphia: John Benjamins. Taylor, Christopher. 2003. “Multimodal Transcription in the Analysis: Translation and Subtitling of Italian Films.” The Translator 9 (2): 191–205. ———. 2014. “Multimodality and Audiovisual Translation.” In Handbook of Translation Studies, Vol. 4, edited by Yves Gambier and Luc van Doorslaer, 98–104. Amsterdam: John Benjamins. ———. 2016.“The Multimodal Approach in Audiovisual Translation.” Target 28 (2): 222–36. Thrift, Nigel. 2009. “Non-­representational Theory.” In The Dictionary of Human Geography, 5th ed., edited by Derek Gregory, Ron Johnston, Geraldine Pratt, Michael Watts and Sarah Whatmore, 503–5. London: Wiley-­Blackwell. van Leeuwen, Theo, Emilia Djonov, and Kay L. O’Halloran. 2013. “ ‘David Byrne Really Does Love PowerPoint’: Art as Research on Semiotics and Semiotic Technology.” Social Semiotics 23 (3): 409–23. Vanoye, Francis. 1985. “Conversations publiques.” Iris 3 (1): 99–188. Vietnamese Cinema Blog. 2015. “Happy Mother’s Day from Surname Viêt Given Name Nam by Surname Nam Given Name Nam.” Accessed June 20, 2018. www.vietnamesecinemablog.com/2015/05/10/happy-­mothers-­day-­from-­surname-­viet-­g iven-­name-­nam-­by-­ surname-­nam-­given-­name-­nam/. Wu, Qunfang, Yisi Sang, Shan Zhang, and Yun Huang. 2018. “Danmaku vs. Forum Comments: Understanding User Participation and Knowledge Sharing in Online Videos.” Proceedings of the 2018 ACM Conference on Supporting Groupwork, 209–18. Accessed April 20, 2019. https://dl.acm.org/citation.cfm?id=3148344. Xu, Yizhou. 2016. “The Postmodern Aesthetic of Chinese Online Comment Cultures.” Communication and the Public 1 (4): 436–51.

Filmography Boyle, Danny. 2008. Slumdog Millionaire. IMDb entry. www.imdb.com/title/tt1010048/? ref_=nv_sr_1?ref_=nv_sr_1. Kosminsky, Peter. 2017. The State. IMDb entry. www.imdb.com/title/tt6068620/?ref_ =fn_al_tt_2. Minh-­ha, Trinh T. 1989. Surname Viêt Given Name Nam. IMDb entry. www.imdb.com/title/ tt0098414/. Sorrentino, Paolo. 2008. Il Divo. IMDb entry. www.imdb.com/title/tt1023490/?ref_ =fn_al_tt_1.

5 TRANSLATING “I” Dante, literariness, and the inherent multimodality of language Matthew Reynolds

1. Adam, language, and the idea of multimodality In Dante’s Paradiso, the character named “Dante” (who is also the first-­person narrator) encounters the first human, Adam. With what may seem to be some lack of tact, Dante addresses Adam thus: “O pomo” (“O apple”) (Alighieri 1975a, 383).Yet Adam does not bridle at being reminded of what one might expect he would most want to forget; in fact, he does not even appear to notice. The characters seem here to inhabit a realm of smooth communication where potential misunderstandings and faux pas evaporate. Like the other souls in Paradise, Adam is at ease with his quondam sin, since now he is wholly in harmony with the divine plan. In line with this paradisal way of being, Adam, who is represented by a hovering light, has the power of reading (as we say) Dante’s mind. He can apprehend for himself the questions that Dante wants to ask, since he sees them reflected in the verace speglio (“true mirror”) of divine knowledge: how long did Adam spend in the garden of Eden? What was the real reason for the Fall? And what was the language or kind of speech (idiöma) that he used? However, in the phrasing of this last question, which must have been conceived by Dante in the second person but is uttered by Adam in the first person, there is a ripple that disrupts the perfect communication we have come to expect: e l’idïoma ch’usai e che fei. And the language (or “speech,” or “idiom”) that I used and that I made (or “created”). (Alighieri 1975a, 384) The ripple is “e che fei.” Dante-­the-­character cannot have known that Adam created the language spoken in Eden, so those words cannot have originated in his

118  Matthew Reynolds

thoughts. If there was something more in Dante’s mind than the question “what language did you use?” it can only have been “and how was it created?” The reason is that this was a live problem in medieval linguistics, and in his earlier book De vulgari eloquentia, Dante had expressed a different opinion: Dicimus certam formam locutionis a Deo cum anima prima concreatam fuisse. We assert that a certain (or “particular”) form of speech (or “language”) was simultaneously created by God along with the first soul (or “person”). (Alighieri 1996, 1. 6) So, there is an energy of correction in Adam’s words: “that I made, that was made by me.” We might read this as an element in the drama of the poem: Adam-­the-­ character puts Dante-­the-­character to rights. Or we might read it in terms of intellectual biography: Dante-­the-­writer revises his earlier view. Either way, what is presented as miraculous and perfect repetition in another mind and medium, the representation of one person’s thought in another person’s speech, turns out to generate significant change. The thought cannot but become different as it emerges in Adam’s words. In the act of uttering the question, he begins to answer it. This moment points towards the sort of perception later articulated by Derrida, that what might seem like mere repetition always involves some element of change: there is always iteration within reiteration (Derrida 1982, 325). It also gives reason for probing the definition of what we nowadays call “modes.” In Gunther Kress’s words, “Mode is a socially shaped and culturally given semiotic resource for making meaning. Image, writing, layout, music, gesture, speech, moving image, soundtrack and 3D objects are examples of modes.” (Kress 2010, 79). But what about thought? Is thought a mode? Thought is certainly “socially shaped,” but it does not seem quite right to call it “culturally given.” The claim that writing or image are inextricable from culture seems plausible, but it is possible to conceive of something that we might want to call “thought” existing independently from anything we might define as culture. Perhaps there are some kinds of thought (such as thinking in words) that are culturally given and other kinds of thought (such as the perception of being sleepy) that are not. So perhaps there are different modes of thought. Perhaps thoughtspeech belongs to the same mode as vocal-speech and is something different from thought that is not verbalised. Is the move from Dante-­the-­character’s thoughts to Adam’s words then an instance of translation across modes, or reiteration within a mode? We could try to argue that if the thoughts have been verbalised in Dante’s mind, then it is more like reiteration within a mode, and if they have not, then it is more like translation across modes, but the truth is that there is no right answer. We cannot draw a solid boundary around what counts as “speech”; therefore, we cannot give a clear definition of mode; and therefore, there is no secure distinction between translating across two modes and reiterating within one mode. Kress has recognised the blurriness in the definition of modes: “there is no straightforward answer to questions such as: ‘Is font a mode; is layout a mode; is colour a

Translating “I”  119

mode?’ ” His way around this problem is to suggest that “what a community decides to regard and use as mode is mode” (Kress 2010, 87). Certainly, observation of a community’s practices and assumptions can help in deciding whether it makes sense to call something a mode. The fact that there are people whose job is graphic design gives support to the idea that “layout” should count as a different mode from “writing.” Equally, the fact that many people are likely to agree that speech is distinguishable from gesture can support the claim that “gesture” and “speech” are different modes. All the same, it is worth insisting that a community does not usually decide to regard anything “as mode,” for “mode” is a technical term. Really, Kress and other theorists of multimodality decide what to regard as mode, and the reason for chopping up the landscape of meaning-­making in the way they do lies in the explanatory power of that act of conceptualisation. Is “handwriting” one mode, “typewriting” another, “word processing” another, “writing online chat” another again? And what about writing “with a biro” vs. “with a fountain pen” vs. “with a quill pen” vs. “with chalk” vs. “with a marker on a whiteboard” vs. “with an electronic pen on a smartphone”? We could argue that these are all different modes, all distinct resources for meaning-­ making that are socially shaped and culturally given for different contexts and purposes. Or we could decide that they all count as “writing,” which is one mode. The choice will be determined by the kind of argument we want to make. There is also the non-­trivial problem of identifying “submodes,” which Marcus Tomalin considers in his chapter for this volume (Chapter 6). The reason why this clarification matters is that it affects what we take “multimodality” to mean. Just like “multilingualism,” “multimodality” does not name a multitude of entities called “modes” that are separate and countable. Rather, it points to an ever-­varying continuum of resources for meaning-­making that can be divided up in different ways. Social practice bunches some of these together so that it can seem obvious they should be defined together as one mode (e.g., “writing”). The same is true of language (e.g., “Italian”). But if you look closer you can always find reasons for separating out the strands and seeing them as different modes (again, the same is true of languages, with their different dialects, idioms, registers, and styles). Back in Dante’s Commedia we find a picture of language, and of its place in the ever-­varying continuum of meaning-­making, that illustrates these points. As Adam continues his disquisition in Paradiso 26, he explains that the language he spoke in the garden of Eden had completely disappeared even before the catastrophe of the Tower of Babel (traditionally, of course, the moment at which language variety began). The reason flows from the fact that he alone had created that language: every product of human intelligence (“effetto … razïonabile”) is subject to change because human pleasure or choice (“piacere”) continually refreshes itself just like the sky (“rinovella/seguendo il cielo”). He gives an example: Pria ch’i’scendessi a l’infernale ambascia, I s’appellava in terra il sommo bene onde vien la letizia che mi fascia;

120  Matthew Reynolds

e El si chiamò poi: e ciò convene, ché l’uso d’i mortali è come fronda in ramo, che sen va e altra vene. (Alighieri 1975a, 384) Before I went down to the infernal stifling-­suffering,/ I was the name on earth of the highest good/ whence comes the happiness that swathes me;/ and EL he was called later: and that’s the way it has to be/ because the work of mortals is like a leaf/ on a branch: it goes and another comes. Since Adam is right about language-change, any piece of language might be taken to exemplify the processes of reiteration and alteration to which he refers: they are all-­pervasive. But the texture of this passage, just like Adam’s voicing of Dante’s thoughts, seems designed to bring their variety and complexity into the light. There is the way the two words for God are hooked into the phonetic and alphabetic weave of the lines, one finding an echo and reflection in the line before, and the other in the line after, in a concatenation of repetition-­with-­difference: Pria ch’i’scendessi a l’infernale ambascia, I s’appellava in terra il sommo bene . . . e EL si chiamò poi: e ciò convene, ché l’uso d’i mortali è come fronda There is the word fronda which, with its accompanying image, has itself gone and come, from its origin in Horace’s Ars Poetica, through many different appearances with varying connotations throughout the Commedia; as when, 60-­odd lines earlier in this same canto, Dante-­the-­character has described all creatures in all of creation as leaves or fronds that enfrond the garden of the eternal gardener, “le fronde onde s’infronda tutto l’orto/de l’ortolano etterno,” and then described his own reaction at being introduced to Adam as being like a leaf (fronda) that bends in the wind before bouncing back (Alighieri 1975a, 382–83). And then there is the way this passage too represents a change of mind on the part of Dante-­the-­writer, for in De vulgari he had thought that Adam’s language had remained unchanged until the moment of Babel; that it had been Hebrew, and that the name for God was “El” (Alighieri 1996, 1. 4–6). In line with this account of continual change, what might be called “foreign” languages are not marked off as distinct but rather brought together into one flexible linguistic weave. As Gianfranco Contini, Zygmunt G. Barański, and others have observed, Virgil’s Latin, the Latin of the church, Arnaut Daniel’s Provençal, and many dialectal forms all blend with Dante’s mainly Florentine Italian to create one variable tongue which continually shuffles registers and genres (Contini 1970, 171; Barański 1986, 8). Translation between what might be thought of as the separate modes of speech and writing likewise figures more as a continuum than a leap. The poem focuses

Translating “I”  121

on moments where movement morphs into sound, and sound into writing, so that its own processes of finding words (both oral and alphabetic) are layered into an imagined universe of people and objects that are continually themselves becoming language. There are many instances that we could point to: the moments in Paradiso when the shining lights of the blessed souls sing and dance themselves into significant shapes: an eagle of justice, a river of life, or letters of the alphabet. The episode in Purgatorio 10 where Dante views bas reliefs so brilliantly done that his mind’s ear can hear the figures speak. Inferno 26, where the flame in which the soul of Ulysses is punished flutters back and forth like a speaking tongue until at last it flicks out words. Or Inferno 13, where Dante and Virgil are in a dark, knotty, thorny wood. Dante reaches out his hand a little (“porsi la mano un poco avante”) and gathers a little branch (“colsi un ramicel”). A voice leaps out: “Why do you snap me?” (“Perché mi schiante?”). From the break come words and blood together, as from a green stick that is burning at one end so that the other end groans and whistles as the air blows out (“geme/e cigola per vento che va via”). Dante, startled, drops the twig and becomes “as the man who fears” (“stetti come l’uom che teme”) (Alighieri 1975a, 54). We could draw on the theoretical work of Nelson Goodman, in Languages of Art, and W. J. T. Mitchell, in Iconology, to gloss this episode. As they show, the distinction between “linguistic” and “non-­linguistic” signification is hazy and cross-­hatched with continuities. Signs cannot be separated into the neat categories of “icon,” “index,” and “symbol”; rather, “the differences between sign-­types are matters of use, habit and convention” (Mitchell 1986, 56–69). So, in Inferno 13, something that we might choose to call an “index” of pain – the flow of blood and breath together – develops seamlessly through whistling and groaning into sounds and letters which have “symbolic” meaning as words (“Perché mi schiante”), though they also, in their own patterning, offer an “iconic” picture of the sound of the stuttering branch: “ché ... chi.” Dante’s reaction is an “index” of something frightening, but also an “icon” of fear, and one that can be readily evoked by “symbolic” verbal signification. Bodies blur into body-language, and that language is a mix of oral suggestion and visual patterning. A sequence of letters towards and across a line-­end draws a diagram of the casual, hesitant advance of Dante’s hand towards the branch: “porsi la mano un poco avante/e colsi”; but the letters, of course, also evoke sounds, and the feel of the sounds inhabiting a mouth with a moving tongue and lips, and perhaps an accompanying gesture, most evokes the physicality of the movement that is being described. Recent neuroscientific investigations of reading have recognised the complexity of the relationship between sight and sound in language. In reading, we can harvest meaning from silent words without having to pronounce them mentally, but the implied sound always has the potential to be activated, sometimes in harmony and sometimes in conflict with the spelling (Dehaene 2010, 26). Dante’s text provides rich illustration of the multimodal gradation which always traverses language when it has a written as well as an oral form: from the visual to the pictorial to the patterned to the written; from writing to imagined speech to uttered speech to noise.

122  Matthew Reynolds

2. Practices of translation and the multimodality of language As language is not one simple, distinct mode, so translation is not one simple, distinct activity. Rather, the inherent and complex multimodality of language gives rise to a great variety of translation practices. At the macro level of professional training and employment, there is a broad distinction between the translation of speech, usually called “interpreting” in English, and the translation of writing, usually just called “translation.” This established and practical division of labour is complicated by many transmodal practices. Subtitling (discussed in the previous chapter) is the most prominent of these; others occur when simultaneous interpreters mug up expected words, phrases, and lines of argument from written documents in preparation for their oral performances, or when the speech of consecutive interpreters in legal contexts is taken down to form a written record. On the other side of the broad distinction, translators of writing, especially in the literary sphere, often take account of the implied sonorities of the texts they read and recreate, while translators of plays or political speeches produce written texts that are designed for speech. Poetry can mingle writtenness and spokenness in complex ways, as in a recent performance of poems by the German poet Ulrike Sandig, at which the poet spoke lines in both German and English into a sound box, so that they recurred on a loop, with more language being spoken over them, and Sandig’s translator Karen Leeder at times chiming in too, in a vivid oral expansion and mutation of written parallel text.1 The field of written translation is also haunted by speech in the shape of metaphor. Good translations are often described in terms of voice, as when Antoine Berman praises successful translations of Sappho as “des poèmes rendus parlantes,” or when John Dryden famously aimed to make Virgil “speak such English, as he wou’d himself have spoken, if he had been born in England, and in the present Age” (Berman 1999, 84; Dryden 1987, 330–31). These images of voice imply a degree of what is often called “freedom” on the part of the translator. They are generally contrasted with translations that are described in terms specific to writing, and are taken to be more “close”: translations that are “literal” (i.e., literally, “to do with letters”) or “word for word” or “line for line” – formulations which suggest the measuring out of bits of written text. In harmony with the widespread phonocentric assumptions traced by Derrida, the “freedom” of a metaphorical appeal to “speech” in written translation is typically presented as a higher kind of fidelity. By hearing the author’s “voice” and imitating it in your translation – the idea is – you are reaching for a deeper sort of meaning than can be gathered from attention to the words as they are printed, the “mere words” as they were called by a Victorian translator, F. A. Paley (Aeschylus 1864, v.). An alternative contrast, between “letter” and “spirit,” enters into the same pattern of metaphorical suggestion, as “spirit” is derived from the Latin for “breath.” No doubt, “free” translations are sometimes easier to read out loud than “literal” ones. All the same, it is important to see that in this wide field of kinds of translation there is no simple gradation from the more visual modality of the written to the more oral modality of the spoken. The metaphors associated with speech and writing

Translating “I”  123

do not line up with the presence of physical spokenness or writtenness in a given translation practice, and the elements can combine in surprising ways. What we are faced with here is not a contrast between attention to “writing” on the one hand and “speech” on the other, but rather a multitude of varying practices of interpretation, all of which are multimodal. For instance, when Browning translated the Agamemnon in 1877, he aimed to produce a “transcript” of the Greek text from which all hearable “magniloquence and sonority” had been lost, leaving only “the very turn of each phrase in as Greek a fashion as our English will bear.” To Ezra Pound the result was “stilted unsayable jargon” (Reynolds 2013, 153).Yet to Tony Harrison, whose translation of the Oresteia became one of the most compellingly sonic theatre experiences of the 1980s, Browning’s text seemed full of noises:“the seeds of my principal choices,” he said, “were lurking there in Browning from the beginning” (Reynolds 2011, 53). The extreme instance of this ambushing of the literal by the phonetic is the word-­for-­word cum voice-­for-­voice translation of Catullus by Louis and Celia Zukofsky which, as they explained, “follows the sound, rhythm, and syntax of his Latin – tries, as is said, to breathe the ‘literal’ meaning with him,” as when they translate this line from poem 70: in vento et rapida scribere oportet aqua into this English: in wind o wet rapid a scribble reported in water. (Catullus 1969, n. p.) We might see this as the most “written” form of translation, with letter after letter being transcribed, or we might hear it as a primarily oral enterprise, the Latin being pronounced and mimicked in English sounds, which are then recorded on paper. In fact, both processes are running at once and inseparably. It is no less true that a translator who aims for “speech” must also attend to visual structuration. Dryden gave at least as much attention to the patterns of Virgil’s verse as to its imagined intonations. And in practice, of course, he did not make Virgil speak at all, but rather made him write such English as he would have written if he had been born in England and in the present age. As I have argued elsewhere, building on work by Derrida, Naoki Sakai, Robert Young, and others, we need to take account of the ever-­varying continuum of language when we think about translation. The idea that translation transfers something called “meaning” between separate “languages” is a huge simplification – I call it “Translation Rigidly Conceived” (Reynolds 2016, 18). This notion, or “representation” of translation, as Sakai describes it, collaborates with the state-­sponsored processes of linguistic standardisation which lead people to believe that there is a language called (say) “English,” which is both internally homogeneous and separate from other languages (say “French”), and that these separate languages can be lined up to reveal their parallel meanings, as in the image presented by a bilingual dictionary (Reynolds 2016, 18–23; Reynolds 2019). In fact, the communicative

124  Matthew Reynolds

elements and practices that can count as English overlap and intermingle with those that can count as French and other languages. What is too simply called translation “between languages” is in fact translation “across language”: it involves using materials from a different point on language’s ever-­varying continuum to create a new text with new meanings. These new meanings, however, will be sufficiently similar to the meanings of the first text to count as the same for some purposes: to enable us to say that we have read Proust when in fact we have read a translation of Proust, or to understand that we are passing through what we would call “customs” when really it is “douane.” Translations are texts which, though obviously different from their sources, are in some ways used as though they were the same. This is how we can distinguish translations from, on the one hand, printed copies of a book (which are the same for almost every purpose) and, on the other, adaptations and derived or related works which, though they might be similar to the source, are not usually used as though they were the same. Attention to multimodality harmonises with this account and adds some complications. It helps us to keep in mind that translators may be caught, not only between the time-­honoured binary of “form” and “content” but among many meaning-­creating aspects of the spoken and/or written text: tone of voice (actual or imagined), pace, rhythm, layout, visual structures, patterns of sound, font, gesture, timbre, handwriting, and so on. These considerations may seem more relevant to literary translation – especially the translation of poetry – than to other kinds. Yet the complexities that flourish openly in literature, as we have seen with Dante, are latent in all language use. Any translator or interpreter will encounter some of the challenges that literary translators confront in concentrated form.

3. The case of Dante’s “I” Even though translation always operates with and among multimodality, a radically multimodal sign can still cause a translational short circuit. Indeed, it may well be because of translation’s pervasive multimodality that this occurs. When challenged by a radically multimodal sign, complexities that normally muddle along happily in the shadows are forced out into the light, where the established idea (or “representation”) of translation clashes sharply with the reality of meaning-­making practices. This is what happens with the “I” that Dante writes for Adam. Here it is once more: Pria ch’i’scendessi a l’infernale ambascia, I s’appellava in terra il sommo bene There is no authorial manuscript of the Commedia. However, the early witnesses were authoritatively collated in 1966–7 by Giorgio Petrocchi, building on work published by Edward Moore back in 1889, and they concur in asserting that what Dante must have written is a vertical – or perhaps slightly slanting – line (Alighieri

Translating “I”  125

1966–7, 1. 245; 4. 440). But what did it mean? Was it a letter or a numeral? If a letter, which one, and why? Perhaps it wasn’t exactly a letter or a numeral at all? Dante’s recent editor, Anna Maria Chiavacci Leonardi, lists some of the possibilities that were explored by early commentators: “I” might be a single-­letter noun, chosen for its extreme simplicity (“massima semplicità”), which is comparable to that of the divine, or else a number corresponding to divine unity. She prefers another option: that Dante was adopting one of the names of God listed by St Jerome, that is “Ia,” understood by medieval lexicographers as a consonantal “I” pronounced “Ia” but written with its single initial letter (Alighieri 2015, 731). This idea had been exhaustively documented in an article by Gino Casagrande, who pointed to the incorporation of “Ia” into “alleluia” as a supporting factor (“alleluia” is much chanted in Paradiso) (Casagrande 1976). Robert Hollander, in a long note in his edition, noted the further possibility – first suggested by Ernesto Trucchi in 1936 – that “I” might stand for “Iesus” (written forms approximating to the English “I” and “J” are the same letter in Italian) (Hollander 2007). The various interpretive possibilities left their marks on the manuscripts. The vertical might be written in a form that is unequivocally a letter, as in the C14th “Codice 076 (88)” in Cortona which gives a lowercase “i.” Or it might be represented by the letters that form the sound of the number 1 in Italian, “un,” as in “Codice 304 (Riccardiano 1010)” in Florence. Another Florentine manuscript, “Codice 321 (Riccardiano 1035),” which was made by Giovanni Boccaccio, has “un” in the main text, with “I” added in the margin ( Alighieri n. d.). These alterations reveal processes of reading and transmission in which accident and interpretation mix. The variant “un” may have arisen because a scribe pondered the conundrum and chose to clarify what he took to be its meaning, but it more likely came about because the copy text was being read aloud so that scribes could write from dictation: Giorgio Petrocchi observes that the Commedia was often reproduced via dictation because it was so popular (Alighieri 1966–7, 1. 94). The reader simply uttered the easiest spoken equivalent to the mark, perhaps without even realising that there were alternatives. Conversely, even if the mark were transcribed in such a way as to become unquestionably the letter “I,” the pronunciation would remain ambiguous: either the simple Italian /i/ (as in the English “if ”), or else something more like /y/ (as in English “yes”) if the reader has the same preferences as Casagrande, Hollander, and Leonardi. These variants are not the only ones. Another possibility was that the line represented the letter “L,” which could stand for the word “El” because their pronunciation was identical. As we have seen, back when he wrote De vulgari eloquentia, Dante did think that Adam’s name for God was “El,” and he makes his character Adam correct that earlier idea here in Paradiso. But this intellectual drama may well not have struck copyists who were inclined to trust authorities and were perhaps unnerved by inconsistency. All the same, someone who reads the vertical line as “El,” and writes the letters “E” and “l” to make that clear, will then have to change the second name for God which Adam mentions, which really is “El.” Many

126  Matthew Reynolds

manuscripts give “El” for the first name and “Eli” for the second. So much effort to avoid the vertical line. But why? Because these copyists and commentators want the vertical (or slightly slanting) line to have a meaning. It is their job to write sense, or to clarify the sense. And so they pull the radically multimodal mark into one system of signification or another: numerals, or the alphabet in the form that was shared across Latin and Italian. And the commentators adduced (and carry on adducing) reams of theological authorities to justify their choice. There is, however, one among them who has taken the force of the radical uncertainty that lies beneath all the ambiguities we have explored. In 1794, the Veronese scholar Giovanni Iacopo Dionisi published a learned compendium entitled De’ Blandimenti funebri o sia delle acclamazioni sepolcrali cristiane (much of it in fact consists of observations on the text of Dante). Chapter two is a list of acronyms and abbreviations: for instance, “A. B. M.” stands for “anime benemerenti” (Dionisi 1794, 15). The entry for “I” lists “innocens. justus. ispes. Istesanus. in. imperante. Imperatore. indictione. junius, o junias. incomparabili, unus (numero)”; it is also pedestalled by an enormous note on Adam’s name for God. Dionisi ridicules both the readings “El” and “Un,” pointing out that the whole weight of what Adam says lies in the fact that the language he once spoke has been lost: it is a “ridicolosa finzione, che il primo Padre abbia parlato secondo la moderna favella” (“ridiculous idea, that the first father could have spoken a still-­existing language”). Dionisi then elaborates on the iconic suggestiveness of the vertical line: in pronunciation, it tends upwards, as though towards the heavens (“all’insù, quasi al cielo”), while its visual form is the most simple of all (“la più semplice di tutte”): it is therefore suited both to express the love of the soul for heavenly things and the simplicity of the supreme being. He adds that it also happens to be the initial letter of various appropriate words such as “imperio (nome caro al Poeta)” (“empire (a term dear to the poet)”) (Dionisi 1794, 18). Dionisi begins to explore the fact that Adam cannot be translating a word from the Edenic language into Italian. Rather, Adam is quoting from that lost tongue. This means we cannot know which writing system the line belongs to, and we cannot know the system of sounds to which that writing system (whatever it is) corresponds. If “I,” in the Edenic writing system, can be a numeral, then there is no reason why it should correspond to the Arabic numeral “1.” It might just as easily be a million, or infinity, or indeed seventeen. And if “I,” in the Edenic writing system, can be a letter, then it could be pronounced like “a,” “u,” “o,” “q,” “s,” “b,” “r” – or in absolutely any way at all. We simply have no idea. Was there even writing in Eden? If not, then we have to imagine Adam speaking an unknown word which Dante represents in the Italian writing system with “I.” But it might equally be that Adam invented speech and writing together, so that as soon as he named the animals, he also knew how to write those names: he will be able to visualise the line as he speaks the word. In this case, the shape of the vertical line will be an intrinsic part of the word that he utters. Right from the very beginning, language will have been radically multimodal, layering speech, writing, picture – and

Translating “I”  127

perhaps an accompanying gesture, too. Perhaps the significance of the sound-­and-­ line, for Adam, had something in common with the speculations of twelfth-­century Sufi mystics, such as Sahl al-Tustari, Muhasibi, and Niffari, about the vertical mark which forms part of the Arabic writing system, the Alif. As Annemarie Schimmel explains, the Alif was felt to be the primary and perfect signifier, from which all other letter-­shapes had been generated by distortion: To know the alif meant, for the Sufis, to know the divine unity and unicity; he who has remembered this simple letter need no longer remember any other letter or word. In the alif, all of creation is comprehended. (Schimmel 1977, 417) The consensus of European and American scholarship is that Dante did not know of these ideas. But since, as Schimmel says, “there is scarcely a popular poet in the Muslim world, from Turkey to Indonesia, who has not elaborated this topic,” it is possible he may have got wind of them (Schimmel 1977, 418). Even if he did not, one can imagine these kinds of thoughts gathering around the vertical line, as Dante’s mind dwelt on that amazing idea, the word for God, in the language of Eden, which had been entirely lost even before the Tower of Babel. As Dionisi recognised, the single stroke of the pen seems to signify beyond the codes of known verbal languages: it is simple and it points upwards – that is enough. Like the pictorial lines described by the artist Deanna Petherbridge in The Primacy of Drawing, it “asserts its abstract, directional and motile qualities” (Petherbridge 2010, 88). I would not put it past Dante to have known that he was creating a sign that would be hard for scribes to copy, so that their texts would participate in the processes of linguistic change that Adam describes, each manuscript a leaf that goes so that another may come.

4. Translating the “I” As a process of re-­writing, translation has continuities both with scribal transcription and with commentary (especially when the commentary is done in another language such as the Latin of many early expositions of Dante). It is often said that “every translation is an interpretation.” However, as I have shown elsewhere, the relation between translation and interpretation is complex (Reynolds 2011, 59–72). Someone offering an interpretation of a literary text, for instance, aims to expound and clarify its meanings, but a translator aims to create another literary text, which will sustain interpretation in a similar way. A word or mark such as Dante’s “I” may be complex to interpret yet simple to translate. But can we really speak of “translating” the “I,” given that – as we have discovered – it is not really “in” Italian, or any known language, in the first place? Here, attention to multimodality can add nuance to our understanding of the complexities of translation. If you think of translation in the naïve way that I call “Translation Rigidly Conceived,” i.e., as taking something called “meaning” out of one

128  Matthew Reynolds

language and re-­inserting it into another, then the “I” cannot be translated. But if (as I suggested earlier) you think of translation as the creation of a very similar text using resources from a different area of the ever-­varying continuum that is language, then of course the “I” can be translated. In this, it is like other proper nouns, to which Derrida drew attention in his essay, “Des tours de Babel” (Derrida 1985). Names like “Babel” or “Paris” do not belong “in” any language, in the way that “oui” belongs to French and “yes” to English, so they cannot be “translated” on the model of Translation Rigidly Conceived. But of course they can be translated if you have a better conception of translation: you just write out the same sequence of letters, in the new location, and perhaps pronounce it slightly differently. In fact, Derrida’s point also holds for many words that are not proper nouns. Is “table” English or French? Even “yes” and “oui” can pop up and be understood in wildly different points of the continuum of language variation. Yet Dante’s “I” is also different from cases like “Babel” or “Paris” because of its flagrant multimodality. As we have seen, it partakes of the modes of “image” and “gesture” as well as of “writing” and “speech.” But this does not put it out of language, or beyond the reach of translation, any more than does its being a proper noun. Rather, it is an extreme case – a dense microcosm – of the iconic suggestiveness that is always latent in language, and is often made explicit in larger-­scale structural features like bullet points or verse form. In considering what to do with the “I,” translators respond to a multitude of multimodal factors, just as they always do. The obvious move is simply to write “I” again. This is what happens in the very first translations, which were done in the Iberian peninsula. In 1427, Don Enrique de Aragón, Señor de Villena, produced a crib translation in Castilian, which is transcribed in the margin of a manuscript of the Italian text. Where Dante wrote “I,” Villena puts “I” (Alighieri 1427). The same was true in Johannis de Serravalle’s Latin translation of 1416–17 and Andreu Febrer’s translation into Catalan in 1429 (Serravalle n.d, 1416–17; Alighieri 1878, 553). However, the fragility of these acts of accurate perception is indicated by the fact that Villena’s “I” has been dropped from the scholarly edition of his work done by Pedro M. Cátedra: it seems that Cátedra was unable to process the mark, just like many of Dante’s copyists and commentators (Villena 2000, 977). Later translations rely on manuscripts or printed editions in which the “I” has already been replaced by something more straightforwardly linguistic and/or numerical. Balthazar Grangier’s French translation of 1597 gives “UN”; Lebrecht Bachenschwanz’s German translation of 1769 gives “der Eine” (Alighieri 1597, 529, 1769, 194).When the enterprise of Dante translation at last got going in English, in the late eighteenth century, the same picture obtained. The first complete English version of the Commedia was by Henry Boyd, a Church of Ireland vicar living in County Down, and was published in 1802. This contained what was also the first Englishing of the lines from Paradiso which interest us: By one mysterious Name the Lord of all Was known, before I heard the awful call

Translating “I”  129

That led me downward to th’accursed clime.  XXVII. ‘To Eli then the nations learn’d to pray: . . . (Alighieri 1802, 3. 305–6) Like Grangier and Bachenschwanz, Boyd must have been using an edition which opted for the reading “un.” Perhaps – in his case – it was the one printed by Giuseppe Comino in Padua in 1727: it gives “UN” – in capital letters – and “ELI” – also in capitals, just like Boyd’s translation (Alighieri 1727, 426). The second complete English Commedia – also the second attempt to English our lines – was by Henry Francis Cary, the Church of England vicar, writer of verse and scholar of languages whose translation was so much admired throughout the nineteenth century that he was buried in Poets’ Corner in Westminster Abbey, with “TRANSLATOR OF DANTE” engraved on the stone. Cary gives: Ere I descended into hell’s abyss, El* was the name on earth of the Chief Good, Whose joy enfolds me: Eli then ’t was call’d. (Alighieri 1819, 3. 239–40) The note signalled by “*” alerts readers to the textual crux: “Some read Un, ‘One,’ instead of El,” but it then claims that the matter is decided by reference to the mention of “El” in De vulgari eloquentia. Cary must have been using one of the rival editions to Comino’s, perhaps that edited by Baldassare Lombardi (Rome 1791), which gives “El” and “Eli,” just as he does, and points to De vulgari in a note (Alighieri 1830, 672–76). It was possible to read a correct text at this time. Dionisi’s edition, which came out in 1796, prints “I” (Alighieri 1796, 162).Yet it does not seem to have made its way into the hands of English translators. Throughout the nineteenth century, “El” dominates: in the German of Karl Ludwig Kannegiesser (1825) and Karl Streckfuss (1854a); in the French of Eugène Aroux (1842); in the Spanish of D. Juan de la Pezuela (1868) and in the English of Charles Cayley (1854b), John Dayman (1865), Longfellow (1867), and Frederick Kneller Haselfoot Haselfoot (1899). A few years earlier than Haselfoot’s translation, in 1889, the Oxford scholar Edward Moore had published Contributions to the Textual Criticism of the Divina Commedia, which, among much else, reasserts the reading preferred by Dionisi, though with a twist: We have here a great number of variants, but all of them seem to be easily accounted for through a misunderstanding of an original “I” or “J,” standing for the Hebrew letter “Jod,” and representing either the “Jah” or “Jehovah” of Ps.lxviii. 4, &c., or still more probably (since Adam says that his primitive language, of which this would be presumably a fragment, was “tutta spenta,” l. 124), the letter “Jod,” regarded as the cabalistic symbol for God. (Moore 1889, 488)

130  Matthew Reynolds

We can see the traces of this interpretation in a new note to the 1908 Everyman’s Library re-­issue of Cary’s translation: “for El . . . we should read J,” as well as in the 1915 version by another Oxford scholar, Charles Lancelot Shadwell, who gives “Jah,” as does Laurence Binyon in 1943 (with the spelling Yah), and Barbara Reynolds, completing the famous Dorothy L. Sayers’s translation in 1962. But still, no “I.” For that, anglophone readers had to await Charles Singleton’s prose translation of 1975, which took the authoritative text established by Giorgio Petrocchi in 1966–7 as its source: “Before I descended to the anguish of Hell the Supreme Good from whom comes the joy that swathes me was named I on earth; and later He was called El.” (Alighieri 1975b, 297) At once, a new, unshiftable obstruction to translating Dante’s “I” into English becomes evident. The sign that, in Dante’s text, is so ambiguously multimodal, at once writing, speech, gesture and picture, is in English, all too everyday and plain. In Dante, as we have seen, the “I” echoes the shortened form of the Italian “io,” “i,” in “pria ch’i’scendessi” in the previous line: this sounds as a thought-­provoking half-­echo, and/ or a lilt of verbal melody. In English, Adam so much more bluntly refers to himself as “I” and in the same sentence says the name of God is “I.” The static is unbearable. What to do? Allen Mandelbaum, in 1984, chose to place the two “I”s as far apart as possible: Before I was sent down to Hell’s torments, on earth, the Highest Good – from which derives the joy that now enfolds me – was called I; and then He was called El. (Alighieri 1984) Robin Kirkpatrick, in 2007, adopts a contrasting tactic, dialing up the static: Before I sank to Hell’s deep agonies, the Highest Good – from which derives the joy I’m swathed in here – was known on earth as ‘I’. Then afterwards we called it El. (Alighieri 2012, 446–47) An extra “I”-­as-­first-­person-­pronoun is inserted, and placed emphatically at the start of the line which finishes with “I”-­as-­the-­name-­of-­God. Clive James, in 2013, does the opposite: Even before I went down to Hell’s vat Of pain, the Good Supreme that radiates

Translating “I”  131

The endless joy that swathes me now was named, On Earth, in several ways, and that was fit For human usage can no more be blamed. . . (Alighieri 2013, 488) He simply leaves out Dante’s “I,” and ducks the challenge. My own suggestion would be to look for another ambiguously multimodal mark, one which stands in a similar relation to English as “I” does to Italian, which might be a numeral as well as a letter, and is not a word in itself. That mark is “O.” Of course, this would bring in many quite different suggestions from Dante’s “I” – God as zero!? God as an expression of surprise!? – but it is still a picture and implicitly a gesture, as well as a letter and a numeral, and one whose form can be taken as a symbol of perfection. What we can see in all these offerings is the variable terrain across which translation moves, the ever-­shifting continuum of language, which is also a continuum of multimodality. “I” – like all elements of language – is inherently multimodal, in English as well as in Italian. But the ratios of its multimodality are different. In English it is pulled further into the coded area of the language: it is more of an everyday word. This does not make it “untranslatable,” for any translation of anything always introduces some difference: the creation of difference is part of what translation is for. Rather, Dante’s “I” is an especially visible instance of the multimodality that permeates all language, and that therefore affects every act of translation.

Note 1 This event took place at Oxford Translation Day, hosted by the Oxford Comparative Criticism and Translation research centre at St Anne’s College, 8 June 2018.

References Aeschylus. 1864. Aeschylus,Translated into English Prose. Translated by F. A. Paley. Cambridge: Deighton, Bell, and Co. Alighieri, Dante. n.d. www.danteonline.it/italiano/codici_indice.htm. ———. 1427. Comedia Dantis Allegerii florentini in qua tracta de penis et punicionibus viciorum et de meritus et premiis virtutum. Biblioteca Digital Hispánica, MSS/10186. ———. 1597. La comédie de Dante. Translated by M. B. Grangier. Vol. 3. Paris: Vve Drobet. Gallica: ark:/12148/bpt6k8701064j. ———. 1727. La divina commedia già ridotta a miglior lezione dagli accademici della Crusca. Padua: Giuseppe Comino. ———. 1769. Von dem Paradiese. Translated by L. Bachenschwanz. Leipzig: Privately Printed. ———. 1796. La divina commedia. Edited by G. J. Dionisi.Vol. 3: Paradiso. Parma: Nel Regal Palazzo. ———. 1802. The Divina Commedia of Dante Alighieri. Translated by Henry Boyd. 3 Vols. London: T. Cadell Jun and W. Davies. ———. 1819. The Vision; or Hell, Purgatory and Paradise, 2nd ed. Translated by Henry Francis Cary. 3 Vols. London: Taylor and Hessey.

132  Matthew Reynolds

———. 1825. Die göttliche Komödie des Dante. Edited by Karl Ludwig Kannegiesser. 5 Vols in 2.Vienna: Fr Schade. ———. 1830. La divina commedia. With a Commentary by P. Baldassare Lombardi. Vol. 3. Florence: Leonardo Ciardetti. ———. 1842. La divine comédie: enfer, purgatoire, paradis. Edited by Eugène Aroux.Vol. 2. Paris: Blanc-­Montanier. ———. 1854a. Göttliche Komödie. Translated by Karl Streckfuss (German, 1854). Dante Lab. dantelab.dartmouth.edu. ———. 1854b. Divine Comedy. Translated by C. B. Cayley. Vol. 3: The Paradise. London: Longman, Brown, Green, and Longmans. ———. 1865. The Divine Comedy of Dante Alighieri. Translated by John Dayman. London: Longmans, Green. ———. 1867. ‘Longfellow.’ Digital Dante. https://digitaldante.columbia.edu. ———. 1868. La divina comedia de Dante Alighieri:Traducida al castellano. Translated by D. Juan de la Pezuela. Vol. 3. Barcelona:Viuda de Luis Tasso. ———. 1878. La Comedia de Dant Allighier, Tralsada de rims vulgars toscans en rims vulgars cathlans. Translated by Andreu Febrer. Barcelona: D. Álvaro Verdaguer. ———. 1899. The Divina Commedia. Translated by Frederick. K. H. Haselfoot, 2nd ed. London: Duckworth. ———. 1908. The Vision of Dante Alighieri. Translated by H. F. Cary. London: J. M. Dent. ———. 1915. The Paradise of Dante Alighieri. Translated by Charles Lancelot Shadwell. London: Palgrave Macmillan. ———. 1943. Dante’s Paradiso. Translated by Laurence Binyon. London: Palgrave Macmillan. ———. 1962. The Comedy of Dante Alighieri the Florentine. Cantica III: Paradise . Translated by Dorothy L. Sayers and Barbara Reynolds. Harmondsworth: Penguin. ———. 1966–7. La commedia secondo l’antica vulgata. Edited by Giorgio Petrocchi. 4 Vols. Verona: Mondadori. ———. 1975a. La divina commedia. Edited by Giorgio Petrocchi. Turin: Einaudi. ———. 1975b. The Divine Comedy. Translated by Charles Singleton.Vol. 3. London: Routledge & Kegan Paul. ———. 1984. The Divine Comedy of Dante Alighieri: Paradiso. Translated by Allen Mandelbaum. The World of Dante. www.worldofdante.org. ———. 1996. De vulgari eloquentia. Edited by Steven Botterill. Cambridge: Cambridge University Press. www.danteonline.it/. ———. 2012. The Divine Comedy: Inferno, Purgatorio, Paradiso. Translated by Robin Kirkpatrick. London: Penguin. ———. 2013. The Divine Comedy. Translated by Clive James. London: Picador. ———. 2015. Commedia. Con il commento di Anna Maria Chiavacci Leonardi.Vol. 3: Paradiso. Milan: Mondadori. Barański, Zygmunt G. 1986. “ ‘Significar per Verba’: Notes on Dante and Plurilingualism.” The Italianist. 6 (1): 5–18. DOI: 10.1179/ita.1986.6.1.5. Berman, Antoine. 1999. La traduction et la lettre ou l’auberge du lointain. Paris: Seuil. Casagrande, Gino. 1976. “ ‘I s’appellava in terra il sommo bene’ (Paradiso XXVI, 134).” Aevum 50 (3): 249–73. Catullus. 1969. Catullus (Gai Valeri Catulli Veronensis Liber). Translated by Celia and Louis Zukofsky. London: Cape Goliar Press. Contini, Gianfranco. 1970. Varianti e altra linguistica. Turin: Einaudi. Dehaene, Stanislas. 2010. Reading in the Brain: The New Science of How We Read. London: Penguin.

Translating “I”  133

Derrida, Jacques. 1982. Margins of Philosophy. Translated by Alan Bass. Brighton: Harvester. ———. 1985. “Des tours de Babel.” In Difference in Translation, edited by Joseph F. Graham, 209–48. Ithaca, NY: Cornell University Press. Dionisi, Giovanni Iacopo. 1794. Dei blandimenti funebri o sia delle acclamazioni sepolcrali Cristiane. Padova: Stamperia del Seminario. Dryden, John. 1987. The Works of John Dryden. Edited by E. N. Hooker, H. T. Swedenberg, Jr. et al. 20 Vols:Vol. 5. Berkeley and London: University of California Press. Goodman, Nelson. 1976. Languages of Art, 2nd ed. Indianapolis and Cambridge: Hackett. Hollander, Robert. 2007. “Robert Hollander (English, 200–2007).” Dartmouth Dante Project. https://dante.dartmouth.edu. Kress, Gunther. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. London and New York: Routledge. Mitchell, W. J. T. 1986. Iconology: Image, Text, Ideology. London and Chicago: University of Chicago Press. Moore, Edward. 1889. Contributions to the Textual Criticism of the Divina Commedia. Cambridge: Cambridge University Press. Petherbridge, Deanna. 2010. The Primacy of Drawing: Histories and Theories of Practice. New Haven and London:Yale University Press. Reynolds, Matthew. 2011. The Poetry of Translation: From Chaucer & Petrarch to Homer & Logue. Oxford: Oxford University Press. ———. 2013. Likenesses:Translation, Illustration, Interpretation. Oxford: Legenda. ———. 2016. Translation: A Very Short Introduction. Oxford: Oxford University Press. ———, ed. 2019. Prismatic Translation. Oxford: Legenda. Schimmel, Annemarie. 1977. Mystical Dimensions of Islam. Chapel Hill: University of North Carolina Press. Serravalle, Johannis de. n. d. “Johannis de Serravalle (1416–17).” Dartmouth Dante Project. https://dante.dartmouth.edu. Villena, Enrique de. 2000. Obras Completas. Edited by Pedro M. Cátedra. Vol. 3. Madrid: Turner.

6 THE MULTIMODAL DIMENSIONS OF LITERATURE IN TRANSLATION Marcus Tomalin

1. Introduction In his 1828 essay “On the Causes of Popular Opinion,” William Hazlitt admitted that “[t]ill I began to paint, or till I became acquainted with the author of The Ancient Mariner, I could neither write nor speak” (Hazlitt 1828, 312). This is a disconcerting remark. The disjunctive conjunction implies uncertainty about the origins of his mature linguistic ability: either it was learning to paint that triggered it, or else it was his encounter with the poet Samuel Taylor Coleridge (whose celebrated The Rime of the Ancyent Marinere had appeared in Lyrical Ballads in 1798) – or maybe both occurrences were equally influential? His syntax is elusive. Despite these tantalising ambiguities, though, Hazlitt’s statement suggests a close connection among the visual arts, literature, and language. And he knew what he was talking about. Having developed a practical interest in painting while in his teens, he had sojourned in Paris from October 1802 to January 1803, where he examined and copied various works by Old Masters, such as Titian and Poussin, that were on display in the Louvre.1 On his return to England he spent three years as an itinerant portrait painter, before making a career for himself as a writer. Despite his mid-­ life career change from paintbrush to pen, he never lost either his love for, or his knowledge of, the visual arts; and in his celebrated essay “On the Pleasure of Painting” (1821), he reflected upon the differences between the two art forms. Having stated bluntly that he never experienced “much pleasure in writing these Essays, or in reading them afterwards,” his disenchantment with prose composition was placed in stark contrast to his enjoyment of creating visual images: “[o]ne is never tired of painting, because you have to set down not what you knew already, but what you have just discovered” (Hazlitt 1821, 7). Hazlitt’s fascination with the associations and disassociations between literary composition and painting may have been distinctive, but it was far from unique.

Multimodal dimensions of literature in translation  135

Many other writers have perceived close connections among language, literature, and the visual arts – and some were creatively involved in all of these domains. Indeed, Hazlitt’s older contemporary,William Blake, provides yet another conspicuous example. His illuminated books, which include Visions of the Daughters of Albion (1793), Songs of Innocence and Experience (1974), Milton (1804–1810), and Jerusalem (1804–1820), were printed using relief-­etched copper plates that combined words and images. His poetry is therefore inherently pictorial, his letter-­forms calligraphic, and his use of colour, line, layout, and design all contribute to the meaning of each work as a whole (Bindman 2009). As a trained artist and printmaker, Blake was aware of the great traditions of literature that combined linguistic forms with visual design, and he drew inspiration from them. The antiquarian sub-­cultures of Romanticism made some of the older traditions of pictorial representation (e.g., illustrated manuscripts and Renaissance woodcuts) much more familiar, while other kinds of contemporaneous images (e.g., illustrations in children’s books) were becoming increasingly common. And the intertwining of image and text continued to flourish in the ensuing decades. To take just two examples from many: Charles Dickens’ novels were illustrated by artists such as George Cruikshank, Hablot Night Browne (a.k.a. Phiz), and John Leech, while the idiosyncratic English writer Stevie Smith produced doodle-­like sketches that accompanied many of her printed poems from the 1930s onwards (Cohen 1980; May 2015). In their very different ways, therefore, these examples constitute works of literature that mingle, or juxtapose, sequences of words and images (in one form or another) to create multimodal ensembles with perceived aesthetic value. Throughout this chapter, the term “ensemble” will be used in the manner defined in the Introduction (p. 13); that is, it will denote representations or communications that contain two or more modes that have been brought together purposefully to produce a collective and interrelated meaning. Further, the person, or people, responsible for creating the ensemble “orchestrate” it by determining how the constituent parts are combined (though this established jargon becomes problematical when musical examples are considered, since such works can involve “orchestration” of a different kind). From a literary critical perspective, multimodal ensembles are of considerable significance in and of themselves, yet the focus in this chapter will fall primarily upon the difficulties that arise when such literary “texts” are translated. The cautionary quotation marks usefully signal a difficulty that will need to be confronted in the ensuing discussion. In general, I will use words like “text” sparingly (whenever possible), since they can be interpreted as insinuating the dominance of linguistic elements in literary works that combine more than one mode. Sensitivity to such matters is essential since, as will be emphasised later in this chapter, the hierarchical orderings of the modes in specific multimodal ensembles are rather more complex than is often acknowledged, especially when translation is involved. Accordingly, this chapter will outline a framework for approaching the difficult task of translating multimodal works that involve both texts and images, and which can be classified as “literature.” Once again, the cautionary quotation marks here are salutary, for several reasons. They remind us to avoid the dubious, but

136  Marcus Tomalin

time-­honoured, convention of privileging writing over all other modes, but they also recognise that, in the current age of experimental multimodal fiction, audiovisual art installations, and the like, the denotation of “literature” has never been less stable (Gibbons 2012). The importance of approaching these issues cautiously is hopefully self-­evident. While literary translation has arguably received a disproportionate amount of critical attention over the centuries, providing a central concern for theorists ranging from St Jerome to John Dryden to Friedrich Schleiermacher to Eugene Nida to André Lefevre to Jacques Derrida to Lawrence Venuti (Venuti 2000), comparatively little attention has been devoted to how translators handle literary works that combine multiple modes. However, the recent emergence of theories of multimodality has created an urgent need to reconsider which particular processes of transposition and meaning (re)constitution should be classified specifically as instances of translation.2 While this topic can undoubtedly be considered in relation to a wide variety of communicative practices, ranging from the prestigious to the seemingly banal, a focus on works of literature reveals distinctive intricacies that merit especially careful scrutiny. This is partly because certain authors have purposefully sought to destabilise culturally specific modal ontologies that were conventionally presupposed during a given epoch. Consequently, in particular poems, novels, and plays, the boundaries separating image from writing, writing from speech, speech from music, and so on, become faint to the point of invisibility – and this has unavoidable consequences for those who undertake the task of translating those multimodal works of literature. Translation in these situations, therefore, has not infrequently probed the limits of communication and meaning-­making in playful ways, often with inventive brilliance. Consequently, the remaining sections of this chapter will examine in detail some of the challenges posed by the translating of multimodal “literature,” with a particular emphasis on the livre d’artiste. As Tong-­King Lee has emphasised insightfully, this undertaking involves determining the role(s) translation plays in the “interplay of different semiotic plans of representation” (Lee 2012, 242). The specific framework outlined here is not presented as being pristine and exhaustive in every respect, of course – far from it. It is, instead, merely offered as a tentative invitation for others to reflect at greater length upon the kinds of issues this chapter addresses in an unavoidably rough and ready manner.

2. Multimodal literary translation As Gunther Kress and others have stressed repeatedly in recent years, “communication has always been multimodal” (Kress et al. 2001, 2). Whether we are considering letters, diaries, architectural plans, medical textbooks, spontaneous conversations, discussions in sign language, comics, scientific treatises, atlases, TV broadcasts, vlogs, or anything else, some combination of speech, writing, images, gestures, music, and so on is usually involved. Since literature is merely an imperfect umbrella term for a wide range of distinctive forms of communication (e.g., novels, short stories, sermons, essays, sonnets, epic poems), then it follows

Multimodal dimensions of literature in translation  137

naturally that literature has always been multimodal too. Yet there are clearly degrees of multimodality, and some literary works are certainly more multimodal than others. Shakespeare’s Sonnet 1, for example, as printed in the 1609 Quarto, can be subjected to various forms of traditional literary analysis that address issues relating to its form and content: From faire!t creatures we de!ire increa!e,

That thereby beauties Ro!e might neuer die, But as the riper !hould by time decea!e,

His tender heire might beare his memory:

But thou contracted to thine owne bright eyes,

Feed’!t thy lights flame with !elfe !ub!tantiall fewell, Making a famine where aboundance lies,

Thy !elfe thy foe, to thy !weet !elfe too cruell:

Thou that art now the worlds fre!h ornament, And only herauld to the gaudy !pring,

Within thine owne bud burie!t thy content,

And tender chorle mak!t wa!t in niggarding: Pitty the world, or el!e this glutton be,

To eate the worlds due, by the graue and thee.

It was printed using a specific type-­font and a particular set of Early Modern orthographic conventions (e.g., the spelling “fewell” in line 6). Structurally, the text can be divided into three quatrains and a couplet, which follow the rhyme scheme ABAB CDCD EFEF GG. Thematically, it contains allusions to the Eden of Genesis, aesthetics, procreation, and the undesirability of wasteful self-­consumption. These properties and qualities can all be examined via a literary critical response that focuses primarily upon the text as a piece of writing. Since writing is one of the modes conventionally recognised in multimodal analyses, then a consideration of such matters results from a predominantly monomodal reading. Of course, like most written texts, the sonnet can be read aloud or recited from memory, and when that happens the semiotic affordances of speech come into play (e.g., accent, intonation contour, voice type). Therefore, even Sonnet 1, which appeared initially in a silent printed form on the page, has the potential to become a multimodal entity. This foregrounds the relationships that exist between certain modes, and this crucial topic merits more attentive scrutiny than it has so far received. For instance, most written texts can be spoken aloud, and therefore speech is (in some sense) a latent mode since it is potentially available whenever we encounter a sequence of written words.3 Yet the degree of latency varies depending on the modes involved. In order to explore this in a preliminary way, later in this chapter I will define specific sub-­types – namely, proximal and distal modes – and reflect upon how they are involved in the process of translating multimodal ensembles. So, like most written texts, Shakespeare’s Sonnet 1, in its printed 1609 Quarto form, can become a multimodal entity if it is read aloud. But perhaps it was

138  Marcus Tomalin

already multimodal to begin with, even before the mode of speech becomes involved? Any such claim necessarily presupposes some degree of certainty about the particular modes that are recognised and deployed as such by a specific community. For instance, printed written texts which also involve illustrations reveal a purposefully multimodal intention underlying their creation, and such examples have proliferated dramatically in recent years, partly because writing conventions have changed due to pervasive technological advances. An interactive work of online e-­literature such as María Mencía’s El Winnipeg: El barco de la esperanza (2017) intentionally combines images (e.g., maps, photographs), writing (e.g., texts by Pablo Neruda, extracts from letters, moving words/characters), and so on. It takes the form of a multimodal ensemble to tell the story of the SS Winnipeg, the ship that arrived in Chile in 1939 with more than 2,000 Spanish passengers aboard, all of them immigrants fleeing Francisco Franco’s dictatorial regime.4 The meaning of the composition results from the interplay of these different multimodal features. They combine, and can be combined, to enable the reader(s) to (re)constitute a range of meanings. It is trivially apparent that a work of this kind is inherently far more multimodal than the Quarto text of Shakespeare’s Sonnet 1, but this is an important point nonetheless. As will be shown later, the multimodal dimensionality of any given source ensemble can be altered by a process of translation, with the result that the target entity has a different multimodal dimensionality compared to the source. This phenomenon has rarely been explored at length in previous studies, but it will provide the primary focus for this chapter. Given this overarching thematic concern, it will prove useful to introduce a simple notational system that indicates the multimodal dimensionality of any given object (be it a work of literature or otherwise). For instance, if P is a specific multimodal ensemble – say, an e-­poem that involves writing, images, and music – then P’s multimodal dimensionality can be indicated by a mode-­quantifying superscript: P 3. This notation merely indicates that P is an ensemble consisting of three distinct modes. Therefore, if we have two multimodal ensembles, P3 and Q5, we can easily see that the former has a lower multimodal dimensionality than the latter, and this can help us compare and contrast these ensembles. The usefulness of this notation will become more apparent later in this chapter when the discussion concentrates more directly on the processes of literary translation.

3. Identity and/or difference? It should be apparent, though, that the rudimentary notational system introduced here presupposes (i) that the various modes are all distinct and well-­defined (at least in relation to specific social groups of users), and (ii) that there exists an ontology which specifies the relations among the various modes. In practice, while there is emerging agreement about the sorts of things modes are and the sorts of semiotic purposes they serve (whether in isolation or as part of ensembles), there is usually no consensus as to the specific set of modes that should feature in any given analysis

Multimodal dimensions of literature in translation  139

in relation to any given social group. As noted in the Introduction, while some theorists treat language as a distinct mode, others (especially Kress) view that usage as a misleading conflation of two distinct modes – namely, speech and writing. These disagreements usually have their roots in the many cultural conventions that overtly or covertly influence our thinking about the semiotic affordances of specific modes. For instance, as mentioned earlier, speech and writing are often closely associated with each other in the Western world. In languages such as English, French, Italian, Spanish, and German (to mention just a few high-­profile ones), it is generally possible to establish a one-­to-­one correspondence between their respective constituent units (e.g., a spoken word and a written word; a spoken phrase and a written phrase), and therefore their respective frames share correspondences.5 Even though the contractions and disfluencies of spontaneous speech complicate this a little (e.g., does “gonna” consist of one lexical item or two?), it is still possible to think of speech and writing as being proximal modes. That is, structural mappings from one to the other are well-­established by prevailing mainstream usage conventions in relatively large social groups (e.g., those for whom, say, British English is a first or second language). The same is true, to a different extent, for speech (or writing) and gesture-­based sign language. From a theoretical perspective, it is intriguing that processes of monolingual intermodal translation involving proximal modes have not commonly been classified as translations at all. That is why the reading aloud of a text is not generally considered to be a translation of the source material, even though the meaning of the written words has been (re)constituted using a different mode with different semiotic affordances (e.g., loudness, stress, intonation contour, rhythm, pitch, duration). Indeed, (re)constitutions of meaning involving proximal modes are usually only considered instances of translation proper if (i) they are language-­related and (ii) they involve a change of language system.6 For instance, if a printed British English poem is read aloud in Metropolitan French, then, in common usage at least, the spoken version would uncontroversially be classified as a translation of the written source. Despite the added bilingual dimension, the modes involved in this case are still proximal: there are broadly accepted ways of mapping written British English to spoken Metropolitan French, even though various options would be available to the translator at many points (e.g., could be translated into spoken French as /tʁis.tɛs/ or /ʃa.ɡʁɛ/̃ , depending on the context).7 By contrast, there are few established conventions for mapping units of speech to units of image. There is no well-­established correspondence between, say, the phoneme sequence /aɪ laɪk ðɪs  ˈpəʊɪm/ and the grammar of visual design (i.e., line, colour, shape, texture, composition), excluding the visual design of written type-­fonts, of course. This is why if 100 people were asked to draw a picture to convey the meaning of the IPA-­notated utterance given earlier, they would produce 100 very different pictures. Indeed, several studies of ekphrasis over the years have emphasised that images do not have framing counterparts in spoken utterances that have been established by long-­standing conventions of usage (and vice versa), and this is why speech and image can be classified as distal modes.8 Specifically, unlike proximal modes, the

140  Marcus Tomalin

structural mappings from one distal mode to the other are not well-­established by prevailing mainstream usage amongst particular social groups. Curiously, as Helen Julia Minors notes in her chapter for this book (Chapter 7), (re)constitutions of meaning involving distal modes (e.g., a written poem reworked as a ballet) have not traditionally been classified as instances of translation either. The term has most often been used metaphorically, implying that choreographing a ballet based on a poem is like the process of translation. This situation is further complicated by the fact that an unavoidable relativism characterises the use of modes. As Kress has put it, “[w]hat a community decides to regard and use as mode is mode” (Kress 2010, 87). Consequently, a specific group, within a particular culture, may start using X (where X could be certain colours or sounds, or whatever) in such a way that it functions as a mode as far as the members of that group are concerned, even if no other communities regard and use it in the same manner. Therefore, the precise answer to the important question “is X a mode?” will depend entirely upon the community being considered. Sometimes the right answer will be “no,” and sometimes it will be “yes.” Clearly, practices that emerge in the social activities of one group can, in some cases, spread to other groups, until they become more widely adopted. The situation is dynamic and fluid, never static and rigidly fixed. And, of course, the fundamental underlying assumption that all modes are discretely separable from each other, and therefore can be unproblematically counted using positive integers, is little more than a crude simplifying abstraction. For instance, as discussed in more detail later, speech and music might appear to be obviously distinct modes, but singing and speech are not always easily distinguished from each other (e.g., sprechgesang). Consequently, Derrida’s provocative methodological question is especially pertinent for theories of multimodality: “[s]hould one save oneself by abstraction or save oneself from abstraction?” (Derrida 2002, 42). Recent multimodal theorising about fonts illustrates these categorisation problems well. Most French readers would consider Poems 1 and 2, which are “both” by the avant-­garde French poet Guillaume Apollinaire, to have exactly the same meaning:

Poem 1: Le Dromadaire Avec ses quatre dromadaires Don Pedro d’Alfaroubeira Courut le monde et l’admira. Il fit ce que je voudrais faire Si j’avais quatre dromadaires.

Poem 2: Le Dromadaire Avec ses quatre dromadaires Don Pedro d’Alfaroubeira Courut le monde et l’admira. Il fit ce que je voudrais faire Si j’avais quatre dromadaires.

These poems share the same spellings, the same punctuation conventions, the same line breaks, the same capitalisation patterns, the same semantic connotations, the same syntax, the same lexis, and so on; hence the conclusion that they are effectively identical. The only differences are the style and size of their respective fonts (i.e., Times New Roman, 11pt for Poem 1, and Calibri, 14pt for Poem 2).

Multimodal dimensions of literature in translation  141

Even though these features could easily form part of the bundle associated with the mode referred to as writing (i.e., any printed written text will necessary use specific font types and sizes), most French readers would place minimal importance on such things, to the extent that they would essentially ignore them. By contrast, a community of graphic designers might share both established regularities of usage and assumptions regarding the meaning content of different fonts, and therefore, for that specialist community, Times New Roman and Calibri might well convey consistently distinct meanings.9 Indeed, some fonts have acquired relatively stable semantic connotations that now extend beyond the confines of a small group of professionals with particular knowledge and expertise. Fonts such as Helldorado, for instance, share typological features with several others that were widely used in title and credit sequences in Western films from the 1930s onwards. Films such as Destry Rides Again (1939), Stagecoach (1939), Last Train from Gun Hill (1959), The Good, the Bad and the Ugly (1966), and many many more, all used typographical conventions derived from posters printed in the so-­called Wild West from the mid-­to late nineteenth century. Therefore, the two following sentences have different (self-­ referential) meanings, even for many readers who have no specialist knowledge either of typographical conventions or of Westerns:

Sentence 1:

This sentence makes me think of Clint Eastwood. Sentence 2:

The usage-­related conventions of fonts such as Helldorado have become so strongly established that virtually any letter-­forms with horizontals thicker than the verticals now convey some kind of (vague) notion of Westerns, as a cultural genre. Consequently, while Sentence 1 seems inexplicable to most people – why should that particular sequence of words prompt thoughts about the actor and director Clint Eastwood? – Sentence 2 makes much more sense: it suggests “Clint Eastwood” because the text is written in a font similar to those used in many of Eastwood’s most famous cowboy films. Recognising that certain letter-­ forms can convey meanings in this way, Theo van Leuwen, Nina Nørgaard, Frank Serafini, Jennifer Culson, and others have recently argued that font should be recognised as a mode in its own right (van Leuwen 2006; Nørgaard 2009; Serafini and Culson 2012). And a greater attentiveness to these issues has non-­trivial consequences for the study of literature. As Nørgaard in particular has pointed out, “there is a general tendency in literary criticism to disregard the semiotic potential of typography in literature by focusing monomodally on word-­meaning only” (Nørgaard 2009, 141). This

142  Marcus Tomalin

pervasive disregard reflects the logocentric tendencies of certain literary critical traditions, and it is an attitude that multimodal approaches inevitably challenge and destabilise. Since different publishers have different house styles, we become inured to small variations in the letter-­forms in the literary texts we read, and we assume that such changes have no significant impact on the meaning of the printed materials. Consequently, if fonts are indeed one of the features bundled together to create the mode known as writing, then they are generally deemed to be ignorable ones, even though that stance is evidently a precarious one in certain cases. Nonetheless, Poem 2 would not usually be classified as a translation (either a transduction or a transformation, to use Kress’s terms) of Poem 1.10 The meaning-­material created by the writing has not been moved about or (re)constituted in any way. However, if font were to be recognised as a mode in its own right (for a given community, at least), then a mode-­related change has clearly taken place, and therefore some kind of transposition of the meaning-­making resources has occurred. If that were the case, then Poem 2 would presumably constitute an intramodal translation (i.e., a transformation) of Poem 1, even though this would stretch the traditional denotation of the term “translation” to its breaking point. Similar complications arise when fonts become difficult to separate from the mode image. Ornate, decorative, calligraphic embellishments known as “flourishes” produce written texts that make highly stylised use of colour, contour, line, and composition (Hildebrandt 1995; Lupfer 2003).When considering texts of this kind from a multimodal perspective, it is difficult to determine where writing ends and image begins. Once again, when such entities are created, the precise ontology of the modes often becomes opaque and/or indistinct. Nonetheless, these considerations emphasise strongly the extent to which the multimodal dimensionality of any ensemble (and especially a work of literature) is necessarily conditioned on the meaning-­making practices of a given community. Consequently, to extend further the formalism introduced in the previous section, we can state that the entity P has a multimodal dimensionality of 3 in relation to the specific community c1, and this can be notated as P c31 . If the same object were considered in relation to a different community, c2, and if only two of the three modes were regarded and used by the members of c2, then we can capture this with the notation P c22 . In other words, the multimodal dimensionality of an ensemble can only be determined in relation to a specific social group, and that may often have an impact on the analysis adopted. It also provides a more rigorous framework for exploring the kinds of communicational complexities that can arise when members of c1 and c2 interact concerning multimodal ensembles such as P, for instance via a process of translation. This topic will be considered at much greater length later in the next sections.

4. Translating texts and/or images By contrast with the complexities that characterise the relationship between Poems 1 and 2, it is much less outlandish to assert that Poem 3 is a translation of one of them (or both):

Multimodal dimensions of literature in translation  143

Poem 3: The Dromedary With his four dromedaries, Don Pedro of Alfaroubeira Went all around the world, admiring it. He did the very thing I would have done If I had four dromedaries. Here the written French text of Poem 1 has been converted into a written English text; therefore, intramodal translation has occurred in the familiar way. As mentioned already, this is the kind of monomodal meaning (re)constitution that we traditionally think of whenever translation is discussed. It is, in essence, the prototypical kind of translation. Although they both use the mode of writing, the fact that two languages are involved means that Poem 1 (say) and Poem 3 differ in terms of lexis, syntax, metrical structure, and the like; therefore, they deploy different language systems – so far, so platitudinous. But this conventional scenario becomes more complex when the first printed edition of Apollinaire’s poem is considered. It was published in a collection of short verses entitled Le Bestiaire ou Cortège d’Orphée (1911), and it looked like this:

FIGURE 6.1 

Le Dromadaire

144  Marcus Tomalin

The woodcut images that accompanied Apollinaire’s concise poems were all created by the French Fauvist artist Raoul Dufy. The volume as a whole constitutes a livre d’artiste, and it emerged from a collaboration involving the writer and the visual artist. In this sense, it was not atypical of Apollinaire’s oeuvre. He had already collaborated with the painter André Derain for his prose poem L’Enchateur pourissant (1909), and his abiding interest in the interplay between writing and image has been scrutinised attentively in recent years. In particular, Clive Scott has probed Apollinaire’s fascination with handwriting, doodling, drawing, watercolours, alphabets, scripts, and signatures, noting that “the calligraphic and the graphic live in easy intercourse with one another” in his literary creations (Scott 2015, 21). As a typical work of Apollinarian literature, therefore, “Le Dromadaire” is conspicuously and intentionally multimodal. It combines writing and image (not to mention typeface and the proximal mode of speech). And the placing of the image in relation to the text is of considerable interest and importance. It appears neither on a separate page nor at the foot of the same page, after the poem. Instead, it occurs between the title and the first line of verse. As a result, in terms of layout, the image occupies the same physical space as the poem text, and this emphasises the extent to which writing and image are combined in this particular work. Nonetheless, despite having a multimodal dimensionality of 2 (at least), translations of the poem usually concentrate on the written text and simply provide a version of that in the target language, implying that the two modes used in the original can be harmlessly disaggregated. Consequently, if “Le Dromadaire” is translated into English as Poem 3, then it has a lower multimodal dimensionality than its source material. Crudely, assuming that S (for “Source”) denotes “Le Dromadaire” in Figure 6.1, T (for “Target”) denotes Poem 3, while f1 and e1 denote the communities of (say) Metropolitan French and British English readers, respectively, we can express the effect of the process of translation, →, as follows:11 S 2f → Te11 (1) 1

As (1) indicates, this instance of intramodal translation (i.e., writing to writing) has reduced the multimodal dimensionality of the original poem from 2 to 1: the poem text has been disaggregated from its accompanying image, and the target only (re)constitutes the meaning of the former. For several reasons – some ideological, some practical – this has been the standard practice in literary translation for many years. In his Selected Writings of Guillaume Apollinaire (1971), Roger Shattuck merely presented an English translation of the text with no accompanying image; although advances in modern printing technology have made it much easier now for images to be incorporated into texts, there is still a tendency to separate them (Shattuck 1971). X. J. Kennedy’s 2011 translation of “Le Dromadaire” does includes Dufy’s woodcut, but the poem text (in both French and English) appears on page 20, while the image appears stranded by itself on page 21 (Kennedy 2011, 20–21). There may have been logistical reasons for Shattuck’s exclusion of the woodcuts. The cost and technical difficulty of including images in a printed text was much greater in the 1970s. Also, Le Bestiaire ou Cortège d’Orphée was still subject to copyright, which

Multimodal dimensions of literature in translation  145

may have delimited the available options somewhat. As for the format favoured by Kennedy, it could be argued that such an approach is justified because the writing and the image were produced by different people. While a response of this kind makes rather naïve assumptions about the communal aspects of meaning-­making, it is worth pointing out that exactly the same approach was adopted for many decades in relation to William Blake’s illustrated books (mentioned, briefly, earlier). For instance, his works appeared in writing-­only form even in prestigious editions such as Geoffrey Keynes’s Blake: Complete Writings with Variant Readings (1972), even though the same person had created both the texts and the images in the originals. It was not until good scholarly editions of Blake’s prints became available in the later 1970s that greater emphasis was placed on the relationship between his words and pictures in literary critical studies. As David Erdman noted in the introduction to his epochal The Illuminated Blake (1974), “Blake’s readers are becoming educated spectators also” (Erdman 1974, 13). The process, however, was a gradual one. It is crucial to acknowledge, though, that there are other processes of meaning (re)constitution which could also be classified as forms of translation, and which have different kinds of impact. For instance, consider the alternative approach to the standard form of literary translation exemplified by Poem 3. A contrasting “translation” would separate the woodcut from its accompanying text, and then (re)constitute its meaning by creating a new visual counterpart that uses the semiotic affordances of the image mode in a different way. In this case, we would no longer be dealing with prototypical linguistic translation, since the mode featured in the target would be neither speech nor writing, and no specific language system would be involved. Obviously, a “translation” of this kind would prioritise image (as a mode), and a specific example is given below as Poem 4:12

146  Marcus Tomalin

The images in Figure 6.1 and Poem 4 both depict camels (specifically, dromedaries), but they do so in ways that convey different meanings (at least to readers/viewers belonging to groups with certain shared image-­related semiotic conventions). The fact that the original illustration is a black-­and-­white woodcut unavoidably associates it with the long tradition of relief printing in the Western world. That particular printmaking technique was deployed from c.1400 onwards, and was developed and refined by Old Masters such as Albrecht Dürer and Giovanni Battista Palumba. While single-­leaf images were frequently produced, the method was widely used to provide distinctive pictures in books from the Renaissance onwards, and the technique had been revitalised by Paul Gauguin in the 1890s in works such as his Noa Noa series (created in 1893–94; published in 1901). In an 1897 essay, the painter and etcher Félix Bracquemond had advised that harmony between typography and illustration is essential in a livre d’artiste; therefore, it is better if black-­and-­white are maintained across each page – hence the advantage of using woodcuts (Bracquemond 1897). Influenced by these artistic trends, Apollinaire was convinced by 1909 that “[i]ntimement liée à l’invention de l’imprimerie, la gravure sur bois est celle dont le style se marie le plus heureusement à l’aspect d’un feuillet imprimé” [“Being closely connected with the invention of printing, the style of the woodcut is best suited to the appearance of a printed sheet”] (Apollinaire 1977, 1071). Dufy was knowingly alluding both to the prestigious woodcut tradition and to these more recent aesthetic predilections, when he created the images for Le Bestiaire ou Cortège d’Orphée – though his approach has more in common with Renaissance illustrations than contemporaneous experimental works of French post-­impressionism.13 His image of the dromedary has characteristic features of design and composition. The two-­dimensional anti-­naturalist single-­humped camel is shown in a setting that suggests a desert oasis. Highly stylised palm trees and other forms of vegetation are clearly visible (as they are in many of the other woodcuts in Le Bestiaire). By contrast, the coloured image in Poem 4 was created using a digital camera and subsequent post-­processing. This camel has a greater sense of three-­dimensional modelling (e.g., the shadows on the legs), and it appears against a textured background that could be a rock face. There is no sign of an oasis. Unlike its counterpart in Figure 6.1, this dromedary is wearing a saddle (or a blanket, at least) and a bridle, which suggests domestication and human ownership. While we might be quick (too quick?) to accept that Poem 3 is a translation of “Le Dromadaire,” we would likely experience a much greater sense of disquiet if Poem 4 were to be described as a translation of it as well; this disquiet endures, perhaps, even if we distinguish between “linguistic translation” and “visual translation.” This difference in response is of interest since, at one level of abstraction, the basic processes involved in both cases are identical: S 2f1 → Te11 (2)

Multimodal dimensions of literature in translation  147

Formalism (2) captures the fact that a bimodal ensemble has been disaggregated and then translated monomodally to produce both Poem 3 and Poem 4. Acknowledging this, our inclination to classify the two so differently – i.e., the former constitutes a translation proper; the latter does not – is intriguing, to say the least. Presumably this orthodox stance has its roots in our (logocentric) cultural presuppositions about language and meaning-­making. Although two modes are combined in the original source material, when Figure 6.1 is considered from a literary perspective, we are more likely to recognise that we are primarily encountering a text-­based poem. Therefore, some kind of priority is given to the mode of writing, even though it constitutes only part of the multimodal ensemble. Presumably, if Poem 4 had been titled “Picture 1” instead, then our expectations would have been quite different. This suggests that the readers/viewers/translators of such literary works do not treat each mode in the ensemble as being of equal status. A hierarchical ordering is perceived (whether consciously or subconsciously), and, as a result, one or more modes can become prioritised. While this hierarchical ordering might remain implicit, even imperceptible, when the ensemble is merely being read/ viewed by members of a given community, it often becomes glaringly apparent when it is translated, since the perceived hierarchy partly determines the way in which the meaning of the source is reconfigured as the associated target (i.e., the translation) is created. Michel Foucault drew attention to this aspect of multimodal ensembles in the early 1970s when he reflected upon the relationship between writing and image: ou bien le texte est réglé par l’image (comme dans ces tableaux où sont représentés un livre, une inscription, une lettre, le nom d’un personnage); ou bien l’image est réglée par le texte (comme dans les livres où le dessin vient achever ce que les mots sont chargés de représenter, comme s’il suivait seulement un chemin plus court). Il est vrai que cette subordination ne demeure stable que bien rarement car il arrive au texte du livre de n’être que le commentaire de l’image et le parcours successif, par les mots, de ses formes simultanées. Il arrive aussi au tableau d’être dominé par un texte dont il effectue, plastiquement, toutes les significations. [e]ither the text is ruled by the image (as in those paintings where a book, an inscription, a letter, or the name of a person are represented); or else the image is ruled by the text (as in books where a drawing completes, as if it were merely taking a short cut, the message that words are charged to represent). True, the subordination remains stable only very rarely. What happens to the text of the book is that it becomes merely a commentary on the image, and the linear channel, through words, of its simultaneous forms; and what happens to the picture is that it is dominated by a text, all of whose significations it figuratively illustrates. (Foucault 1973, 39–40; Foucault 1983, 32–3) With its emphasis on the shifting patterns of dominance and subservience, this analysis indicates that multiple stages of transposition and meaning (re)constitution are involved in the process of multimodal (literary) translation. First the author

148  Marcus Tomalin

constitutes a meaning (or meanings) by orchestrating an ensemble (the source), then the translator/publisher transposes the meaning-­making resources, providing an opportunity for the (re)constituting of the meaning(s) by orchestrating a related entity (the target, which may or may not be multimodal) – and then the readers/viewers of the translation (re)constitute the meaning of the target (and/or the source) as they interpret it.14 Crucially, therefore, the translation of multimodal ensembles is a process of transposition and meaning (re)constitution that frequently reveals the modal hierarchy perceived by the translator/publisher. Presented with multimodal source material, one mode may be prioritised, or foregrounded, and this decision may be influenced by an intricate mixture of literary considerations (e.g., genre, style, form) and subjective personal convictions and aptitudes (e.g., greater confidence working with writing rather than with images). Obviously, in cases where an ensemble like Figure 6.1 is converted into a monomodal target text like Poem 3, it is clear that one mode has been prioritised over the other, in a permanent way. The translator’s decision to create a target entity that has (say) a lower modal dimensionality than its source insinuates a perceived hierarchical ordering of the modes. However, to indicate the difference between the processes that produced Poem 3 and Poem 4, the system of notation needs to be modified to indicate the prioritised mode (and, of course, in some cases, more than one mode might be prioritised). If w denotes the mode “writing” and if i denotes the mode “image,” then the two translation processes concerned can usefully be formalised as follows: w

S 2f1 → Te11 (3) i

S 2f1 → Te11 (4) Here the second superscript on T in (3) indicates that the dominant mode is writing, while in (4) it indicates that the dominant mode is image. This formalism now shows how these processes differ, despite being similar, due to the translator/ publisher’s prioritising of one of the modes encountered in the multimodal source. In practice, this prioritising may be entirely subjective, or it may arise overtly from logistical constraints (e.g., copyright restrictions).Whatever the cause, the reduction in the number of modes from 2 to 1 in (3) and (4) means that these processes can both be broadly classified as instances of hypointermodal translation. To explore these ideas further, Figures 6.2 and 6.3 (see next page) both offer examples in which certain aspects of the multimodal source (i.e., Figure 6.1) have been translated, while others have been left completely unmodified. In the case of Figure 6.2, the poem text is entirely unchanged from the source, but the picture of the dromedary has been converted into a completely different target image. This has a significant impact on the meaning of the whole, since the historical and aesthetic connotations of the woodcut image (discussed previously) have been discarded in favour of an image that possesses colour, texture, and quasi-­ three-­dimensional shaping. Conversely, in Figure 6.3, the source image remains

Multimodal dimensions of literature in translation  149

FIGURE 6.2 

New image

FIGURE 6.3 

New text

untouched, but the poem text has been translated from French into English. Once again, in obvious ways, this causes the meaning of the target to differ from that of the source. Assuming, once again, that Figure 6.2 is encountered by a community of Metropolitan French readers, while Figure 6.3 is intended for British English readers, the two translation processes exemplified here can be captured by the following respective formalisms: i

S 2f1 → Te21 (5) w

S 2f1 → Te21 (6) In both of these cases, the multimodal dimensionality of the literary works remains unchanged, but the perceived prioritising of one mode over the other can be inferred by considering those components of the multimodal ensemble that the translator chose to alter. Crucially, it does not follow from this that the prioritised mode (or modes) is necessarily perceived by the reader/viewer as being the dominant or most important one in the resulting ensemble. For instance, a French reproduction of Richard Pousette-­Dart’s ink and watercolour painting Gothic Garden (c.1948) might give the title as Jardin Gothique. In this case, the title would have been prioritised by the translator, assuming that the reproduction of the painting was left entirely unchanged, but most French-­speaking viewers would probably still consider the image to be the most important part of the ensemble. In examples (5)

150  Marcus Tomalin

and (6), both modes are retained in both translations, but in Figure 6.2 the image has been changed by the translator’s activity, while the text in Figure 6.3 has been rendered into a different form. In these cases too, the translator’s prioritising of certain modes does not determine a hierarchical ordering on behalf of the reader/ viewer. As ever, those individuals are free to interpret the multimodal ensemble as they wish, and it is certainly possible (indeed likely) that the relative dominance/ subservience of the modes will shift and change during the process of interpretation (as Foucault recognised).

5. A musical dimension The examples considered so far have only included instances where (a subset of) the modes involved in the source ensemble are also involved in the target. However, other scenarios should be scrutinised, too. For instance, in 1919 Francis Poulenc set six of the Bestiaire poems to music, and the first song in the sequence is “Le dromadaire” (Poulenc 1920). In fact, there are two different versions of the song. Initially, Poulenc scored it for flute, clarinet, bassoon, string quartet (i.e., two violins, viola, and cello), and voice, but he subsequently published a version for voice and piano only. The relationship between either of these versions and Apollinaire’s and Dufy’s poem-­picture merits careful consideration. In Poulenc’s settings, the written text of the original has been disaggregated from its associated image (as usual); a transposition has taken place, and the meaning-­material of the former can now be (re)constituted using the semiotic resources of the mode of music. Immediately, though, some caution is required, since “music,” like “language,” is potentially an implausible mode. If the term is used loosely, with reference to all forms of music – everything from printed scores, to mp3 recordings, to live performances – then arguably it conflates different materialities and logics that should rightly be kept separate. Therefore, at the very least, it makes sense to distinguish between written music and performed music – that is, music as manifest silently by means of a written system of notation on a page (e.g., the score of a string quartet) versus music as manifest in acoustic soundwaves (e.g., an actual performance of a string quartet using instruments). This is similar to the aforementioned distinction between writing and speech, and therefore it goes without saying that close conventional associations of usage connect the semiotic affordances of the two modes of written and performed music. The violinist in a string quartet produces the required acoustic soundwaves by reading the notes on the page and interpreting them by means of physical gestures (e.g., the placing of fingers on the fingerboard; movements of the bow). To use the terminology introduced earlier, written music and performed music are proximal modes, since, like speech and writing, they respectively make available established one-­to-­one mappings between specific structural units (e.g., an E-­flat major chord, a crotchet note, a minim rest, a plagal cadence). Given this initial distinction, we may also wish to subdivide both written and performed music into two submodes – singing and accompaniment – since these are characterised by different affordances. Singing usually involves linguistic elements that are made available by

Multimodal dimensions of literature in translation  151

the text underlay in the printed score (e.g., words, phrases, sentences), and, in specific contexts, it can mingle with the mode of speech (hence the existence of techniques such as “sprechgesang” and “sprechstimme”). This suggests that singing and speech are a proximal mode/submode pairing, since they have an especially close relationship. By contrast, accompaniment usually involves language-­less music (e.g., a piano part), which means that speech and accompaniment form a distal mode/ submode pairing.15 Given all of this, if we focus on a particular live performance of Poulenc’s “Le dromadaire” (and, to simplify matters, specifically the original version, for voice and chamber orchestra), then the song can be viewed as an audiolinguistic intermodal translation (i.e., a transduction) of the source material. If we assume that the community encountering Poulenc’s song is the same French-­speaking community that read “Le Dromadaire” (Figure 6.1), and if p denotes the mode performed music, then the nature of this translation process can be represented as follows: p

S 2f1 → T f11 (7) This example reveals a further limitation in the system of notation being developed here. More precisely, (7) unhelpfully conceals the fact that the mode involved in T is not involved in S at all. Once again, though, the system can be easily modified to indicate when (some of) the modes or submodes in the source and the target are the same and when they are different. Henceforth, the superscript n[x] will be added to the symbol for the target, to indicate that x of the n target modes are also encountered in the source. Using this notation, formulae (1) and (7) can be rewritten as: w

S 2f1 → Te11[1] (8) p

S 2f → T 1f [ 0 ] (9) 1

1

In short, (8) indicates that the source (Figure 6.1) and target (Poem 3) have the mode of writing in common, while (9) shows that the prioritised mode in the Poulenc song (i.e., performed music) is not involved at all in the Apollinaire-­Dufy original (though the submode of singing is proximal to speech, which is a latent mode in Figure 6.1).16 The plus sign (+) associated with the mode p simply indicates that this prioritised mode does not feature in S. As mentioned already, though, it is often a misleading simplification to designate only one mode in an ensemble as having been “prioritised,” since different modes may rise to prominence at different times as the listener (re-­)constitutes the meaning of the target, especially when the ensemble has temporal duration (as in the case of a performed song). To consider one specific instance, in Poulenc’s setting of Apollinaire’s poem text, there are 10 bars of music before the vocal part begins. Therefore, inevitably, the submode of accompaniment is necessarily prominent for the duration of those bars, since no other modes or submodes are involved. However, the hierarchy is potentially readjusted

152  Marcus Tomalin

when the two submodes of singing and the accompaniment occur together, starting from bar 11. Although the examples considered so far have only included instances where the multimodal dimensionality of the target is less than, or equal to, that of the source, other translation processes can produce target ensembles that have a higher multimodal dimensionality. Accordingly, if Figure 6.1 were displayed online in such a way that an audio recording of the Poulenc song played whenever the website were accessed, then the readers/viewers/listeners would encounter the French poem text, the woodcut image, and the musical realisation of the poem text as a single multimodal ensemble. In that case, the multimodal dimensionality of the target would be greater than that of the source. While two of the modes involved would be the same (i.e., writing and image), one would be entirely new (i.e., performed music). This is captured in formula (10): p+

S 2f1 → T f31[ 2 ] (10) This process can be classified as one of hyperintermodal translation, since the mode of performed music has been added to the existing multimodal ensemble. While three modes are involved in this example, performed music has been prioritised by the person who orchestrated the ensemble simply because it has been added to the existing complex. As ever, though, readers/viewers/listeners may perceive very different shifting patterns of modal prioritisation as they read, look at, and/or listen to the webpage. Examples of this kind compel us to confront theoretical issues that have received far less attention than they deserve. As Simon McKerrell and Lyndon C. S.Way have recently reminded us “[m]ultimodal analysis and indeed social semiotic treatments of music have been theorized primarily upon the static and interrelated modes of written text and image” (McKerrell and Way 2017, 8). Given this, any processes that lead to the creation of multimodal ensembles that contain static and dynamic elements ensure that the focus cannot fall exclusively on atemporal configurations. This is just one of the reasons why the study of how translation converts ensembles into ensembles merits serious and sustained attention.

6. Conclusion In his thought-­provoking book about translating Apollinaire, Clive Scott has sought to shift the focus of translation studies away from a preoccupation with the interpretation of the source text towards an exploration of what he calls “the phenomenology of reading.” He urges us to view reading as “a psycho-­physiological experience of text, as an adventure of consciousness and perception in reading, and writing that experience, that consciousness and perception, back into the translation of the [source text].” Crucially, as he points out, an undertaking of this kind “necessitates the multiplication and extension of the linguistic, graphic, and pictorial resources of the translator” (Scott 2015, 18). Viewed in this way, translators are free to present their experience on the page in any manner they see fit.Words need

Multimodal dimensions of literature in translation  153

not be constrained by conventions of syntax; the linear orderings of the source can be reworked in tabular arrangements that convey simultaneity and semantic proliferation; typographical forms can be deployed to suggest performative gestures; photographic fragments can appear as marginal glosses, to supplement the written text and further destabilise conventional linearities and orders. And the writings of Apollinaire provide a rich corpus for developing a theoretical translation framework of this kind – perhaps uniquely so. While the enterprise briefly summarised here could certainly be applied to translation in all its forms, Scott’s abiding concern with literary translation frequently insinuates itself into his vocabulary. His theory deals with scenarios in which translation involves “reading” a “text” and creating a new “text,” prompted by, but not constrained by, the source – and the product of that act of creative reading is expressed primarily by means of “writing,” even though “image” can play a significant role, too. The overarching textual and linguistic emphasis of this formulation may seem a little outmoded in this age of hypertext fiction and animated poetry, yet literature (broadly conceived) undoubtedly merits astute scrutiny when multimodal translation is discussed. This is partly because, over the centuries, numerous writers have sought to destabilise the relationships that exist between the semiotic resources we have come to think of as modes. In his chapter for this book (Chapter 5), for instance, Matthew Reynolds reflects upon Dante’s inclusion of a vertical (or slightly slanting) line as one of the names for God in Paradiso (canto 26, line 134). As Reynolds demonstrates, this line creates difficulties when considered in its literary context. For metrical reasons, it must have some kind of (latent) phonetic form (e.g., it must be associated with at least one phoneme and one syllable), yet because the grapheme “I” is not a conventional part of the linguistic systems of either written or spoken Italian, it is difficult to know which specific form(s) it should take. This complicates the relationship between the proximal modes writing and speech. Indeed, the line seems to have more to do with image than anything else, but how should images be uttered in Italian? And given all of these uncertainties, how should the mystical line be translated? While Dante’s name for God provides a particularly noteworthy instance, it is merely one of many. The witty complexities caused by Corporal Trim’s flourishing of his stick in Laurence Sterne’s Tristram Shandy were alluded to in the Introduction, and, as discussed there, that little detail places an anomalous semiotic burden on the modes involved. It is amusing enough that Trim’s physical gesture is described as being as persuasive on the subject of celibacy as a thousand syllogisms, yet the absurdity is further heightened by the need for the printed page in the novel to represent the physical movement involved (Sterne 1767, 17). Once again, therefore, translation becomes tricky. Do gesture-­imitating squiggles mean the same thing in all languages, or do they need to be re-­expressed in different cultural contexts? Finally, to give just one more example, James Joyce’s Finnegans Wake is a remorselessly experimental novel that causes the modes of speech and writing to mingle and merge extensively. The unconventional orthographies and pervasive neologisms enable Joyce to create a bespoke linguistic system that he characterises as

154  Marcus Tomalin

a “cellelleneteutoslavzendlatinsoundscript,” where “-­soundscript” is at once “Sanskrit” but also simultaneously a compounding of “sound” + “script” (Joyce 1939, 217).17 In punning coinages such as this, orthography and phonology combine to create homophonic possibilities that gesture towards the opportunity of concealed meanings, some of which are revealed when seen, and others only when heard. It is no surprise, therefore, that translating this particular novel has become one of Western literature’s most arduous and forbidding undertakings (O’Neil 2013). With full awareness of intriguing case studies such as these, this chapter has attempted to initiate a conversation concerning some of the ways in which multimodal works of literature can be translated. In particular, the quasi-­mathematical analytical formalism developed in the foregoing sections offers a more precise framework to facilitate the exploration of the impact the translation process can have on the multimodal dimensionality of ensembles. Nonetheless, it is important to ensure that the denotation of the term “translation” does not become so absurdly all-­inclusive as to be effectively meaningless. Accordingly, as a preliminary step, this chapter has advocated much greater specificity about the exact instances of transpositions and meaning (re)constitution being considered in a given case. For instance, translation involving the proximal linguistic modes of speech and writing can be classified as a different sub-­type compared to forms involving (say) music and dance. Greater clarity concerning the ontology of the various (sub)modes would help clarify how different types of translation should be categorised. And the analysis of such things must include nuanced evaluations of the perspectives, strategies, and preferences adopted by the original author(s), the translator/publisher(s), and the reader(s)/viewer(s)/hearer(s). Sometimes, as we have seen, the source ensembles are disaggregated, and certain (sub)modes are discarded; sometimes parts of the ensemble are left completely unmodified, while, on other occasions, new (sub)modes are added to the target, thereby prompting a reconfiguring of the whole ensemble. During this process, certain (sub)modes might be prioritised at specific points, and the act of translation can reveal the nature of these prioritisings. Understanding and elucidating these complexities is a daunting and delicate business, and, as ever, there is extensive scope for a multiplicity of contrasting viewpoints and interpretations. Nonetheless, as this chapter has sought to demonstrate, it is possible to formalise and quantify certain aspects of the multimodal ensembles involved in these fascinating translation-­based scenarios, and, by so doing, the strategies adopted by translators of such source material may become more readily apparent.

Notes 1 See Grayling 2000, chapter 4; Wu 2008, chapter 4. 2 Following Kress’s terminology introduced in the first chapter of this book, I will refer to translation as involving both the “transposition” of the resources for meaning-­making (e.g., a shift from, say, the semiotic affordances of writing to those of dance) and the “(re) constitution” of meaning (e.g., the meaning(s) created by the audience of the resulting dance). However, the added parenthesis in my usage of the latter arise from unease concerning the prefix “re-­”. It is not clear to me when and how the meaning of the source

Multimodal dimensions of literature in translation  155

becomes sufficiently disassembled to require reconstituting. Also, it is not clear to me that the meaning of the target is ever merely a recreation of the meaning of the source. Often it is glaringly obvious that the two are quite distinct – sometimes intentionally so. 3 This closely related to what Matthew Reynolds, in his chapter for this book (Chapter 5), refers to as “thought-speech” (p. 118) – that is, speech that is unspoken: the speech one hears silently in one’s mind as one tacitly reads a written text. 4 María Mencía, “The Winnipeg:The Boat of Hope” (2017): http://winnipeg.mariamencia. com/?lang=es 5 Obviously, this is not the case for all languages since many of them have no written form. See Michael Cahill and Keren Rice, Developing Orthographies for Unwritten Languages (Dallas: SIL International, 2014). As is hopefully apparent, I am using “frame” here in the technical sense – that is, to mark and/or delimit a space, whether material or conceptual, that determines a domain of meaning at a particular level. See Kress 2010, 149–154. 6 The denotation of “language system” is broader than that of “language.” For instance, Geoffrey Chaucer’s The Canterbury Tales (1387–1400) has been “translated” into modern English numerous times (most famously by Nevill Coghill in the 1970s), even though Middle English and Modern English are, strictly, merely different versions of the same language. They can, however, be classified as distinct language systems, since they have different pronouns, different morphological conventions, different syntactic rules, different phoneme sets, and so on. 7 Following well-­established conventions, angled brackets (i.e., < >) denote the orthographical form of a given word, while slant brackets (i.e., //) denote its phonological form. The French words used in this example are and . 8 The wording here is purposefully alluding to the subtitle of Kress and van Leeuwen 2006: The Grammar of Visual Design. The symbols in slanted brackets give the International Phonetic Alphabet transcription of the sentence “I like this poem.” For more information about ekphrasis, see Heffernan 2004. 9 See the discussion in Kress 2010, 87–88. 10 Kress’s terminology was discussed in the Introduction, on p. 5. 11 Unlike Scott, I will use “S” and “T” (i.e., “Source” and “Target”), rather than “ST’ ” (i.e., “Source Text”) and “TT” (Target Text) in this chapter (see Scott 2015, 18). As mentioned earlier, since the focus here is strongly upon multimodal works of literature, I am keen not to allow traditional terminology associated primarily with writing (i.e., “text”) to dominate unnecessarily. 12 In his chapter in this volume (Chapter 1), Kress refers to this phenomenon as “foregrounding” (see pp. 32–33). 13 For a detailed discussion of Apollinaire’s and Dufy’s collaboration, see Read 2013, 17–30. 14 The term “translator/publisher” is used here because, in practice, the work of professional translators is often determined, and even constrained, in various ways by the publishers they are working with. 15 Obviously in certain kinds of choral music (especially a capella pieces) there may be a sung accompaniment, and this may involve linguistic elements. However, such examples can be reasonably classified as involving the submode of singing exclusively. 16 For further discussion of submodes, see Stöckl 2004. 17 The compound noun refers punningly to various members of the Indo-­European language family: Celtic, Hellene, Teuton, Slav, Zend (Persian), Latin, and Sanskrit.

References Apollinaire, Guillaume. 1977. Oeuvres en prose complètes, Vol. 1. Edited by Michel Décaudin. Gallimard: Paris. Apollinaire, Guillaume, and Raoul Dufy. 1911. Le Bestiaire ou Cortège d’Orphée. Paris: Deplanche.

156  Marcus Tomalin

Bindman, David. 2009. William Blake: The Complete Illuminated Books. London: Thames and Hudson. Bracquemond, Félix. 1897. Étude sur la gravure sur bois et la lithographie. Paris: Henri Beraldi. Cahill, Michael, and Keren Rice. 2014. Developing Orthographies for Unwritten Languages. Dallas: SIL International. Cohen, Jane R. 1980. Charles Dickens and his Original Illustrators. Ohio: Ohio State University Press. Derrida, Jacques. 2002. “Faith and Knowledge: The Two Sources of ‘Religion’ at the Limits of Reason Alone.” In Acts of Religion, edited by Gil Anidjar, 40–101. London and New York: Routledge. Erdman, David V., ed. 1974. The Illuminated Blake: All of William Blake’s Illuminated Works with a Plate-­by-­Plate Commentary. Garden City, NY: Anchor Press/Doubleday. Foucault, Michel. 1973. Ceci n’est pas une pipe. Montpellier: Fata Morgana. ———. 1983. This Is Not A Pipe. Translated by James Darkness. Berkeley: University of California Press. Gibbons, Alison. 2012. “Multimodal Literature and Experimentation.” In Routledge Companion to Experimental Literature, edited by Joe Bray, Alison Gibbons, and Brian McHale, 420–34. London and New York: Routledge. Grayling, A. C. 2000. The Quarrel of the Age: The Life and Times of William Hazlitt. London: Weidenfeld and Nicholson. Hazlitt,William. (1821) 1930.“On the Pleasure of Painting.” In The Complete Works of William Hazlitt, Table-­talk,Vol. 8, edited by Percival Presland Howe, 1–18. London and Toronto: J. M. Dent and Sons. ———. (1828) 1933. “On the Causes of Popular Opinion.” In The Complete Works of William Hazlitt, Uncollected Essays,Vol. 17, edited by Percival Presland Howe, 308–13. London and Toronto: J. M. Dent and Sons. Heffernan, James W. 2004. Museum of Words: The Poetics of Ekphrasis from Homer to Ashbery. Chicago and London: The University of Chicago Press. Hildebrandt, Bill. 1995. Calligraphic Flourishing: A new Approach to an Ancient Art. Boston: David R. Godine. Joyce, James. 1939. Finnegans Wake. London: Faber and Faber. Kennedy, X. J., ed. and trans. 2011. Guillaume Apollinaire,The Bestiary, or Procession of Orpheus. Baltimore: The Johns Hopkins University Press. Kress, Gunther. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. London and New York: Routledge. Kress, Gunther, Carey Jewitt, Jon Ogborn, and Charalampos Tsatsarelis. 2001. Multimodal Teaching and Learning:The Rhetorics of the Science Classroom. London: Bloomsbury. Lee,Tong-­King. 2012. “Performing Multimodality: Literary Translation, Intersemioticity and Technology.” Perspectives: Studies in Translatology 21 (2): 241–56. Lupfer, Earl A. 2003. Ornate Pictorial Calligraphy: Instructions and Over 150 Examples. London: Dover. May, Will, ed. 2015. The Collected Poems and Drawings of Stevie Smith. London: Faber and Faber. McKerrell, Simon, and Lyndon C. S. Way. 2017. “Understanding Music as Multimodal Discourse.” In Music as Multimodal Discourse: Semiotics, Power, and Protest, edited by Lyndon C. S. Way and Simon McKerrell, 1–20. New York and London: Bloomsbury. Mencía, María. 2017. “El Winnipeg: El barco de la esperanza.” http://winnipeg.mariamencia.com/?lang=es.

Multimodal dimensions of literature in translation  157

Nørgaard, Nina. 2009. “The Semiotics of Typography in Literary Texts: A Multimodal Approach.” Orbis Litterarum 64 (2): 141–60. O’Neil, Patrick. 2013. Impossible Joyce: Finnegans Wakes. Toronto, Buffalo and London: University of Toronto Press. Poulenc, Francis. 1920. Le Bestiaire ou cortège d’Orphée, poëmes de Guillaume Apollinaire. Paris: Max Eschig. Read, Peter. 2013. “Dufy Draws Apollinaire: Illustration and Commemoration in the livre d’artiste.” In The Art Book Tradition in Twentieth-­Century Europe, edited by Kathryn Brown, 17–30. London and New York: Routledge. Scott, Clive. 2015. Translating Apollinaire. Exeter: University of Exeter Press. Serafini, Frank, and Jennifer Culson. 2012. “Typography as a Semiotic Resource.” Journal of Visual Literacy 31 (2): 1–16. Shattuck, Roger, ed. and trans. 1971. Selected Writings of Guillaume Apollinaire. New York: New Directions Books. Sterne, Laurence. 1767. The Life and Opinions of Tristram Shandy, Gentleman,Vol. 9.York and London: T. Becket and P. A. Dehondt. Stöckl, Hartmut. 2004. “In Between Modes: Language and Image in Printed Media.” In Perspectives on Multimodality, edited by Eija Ventola, Cassily Charles, and Martin Kaltenbacher, 9–30. Amsterdam: John Benjamins. van Leuwen, Theo. 2006. “Towards a Semiotics of Typography.” Information Design Journal 14 (2): 139–55. Venuti, Lawrence, ed. 2000. The Translation Studies Reader, 3rd ed. London and New York: Routledge. Wu, Duncan. 2008. William Hazlitt:The First Modern Man. Oxford: Oxford University Press.

7 TRANSLATIONS BETWEEN MUSIC AND DANCE Analysing the choreomusical gestural interplay in twentieth-­and twenty-­first-­century dance works Helen Julia Minors 1. Translation and multimodality: languages, senses, and cultures Multimodal translation is a highly relevant topic today, in a society where we strive to accept, to learn from, and to engage positively with diversity. The temporal arts of music and dance enable us to share a somatic experience. Three key reasons for the needs to consider translation in the context of the performing arts (especially music and dance) are that: 1 The place of choreomusical studies within institutions is reducing and, as Stephanie Jordan highlights in her recent chapter, it looks as though the “drawbridge is all set to be raised,” further segregating the arts (Jordan, forthcoming). A reassessment of how multimodal translation is a benefit to both creative practice and critical reflection is much needed in order to revitalise this situation. 2 There has been a cultural turn in the arts whereby we recognise that we experience and interpret that experience through metaphor. In other words, in collaborating to co-­create music-­dance works, the artist experiences things in the terms they understand: musicians apply musical understanding to dance and vice versa (Minors 2012a, 2012b). 3 The interaction of different art forms, different cultures, different peoples, is a necessary and important experience which draws out meaning “from dynamic interactions” (Cook 2013, v). The dynamic nature of lived exchanges, I claim, relies on a process of translation. The dialogue between, the exchange across, and the transfer among the artists and their artistic media is a process whereby each strives to understand the other. All interpretative and creative acts have a level of translation (Minors 2013a).

Translations between music and dance  159

Significantly, analogies are created to language. Meaning is constructed (though not only via language but also through the responsive acts of each artistic media). Barriers are present which resist the formation of understanding, as for dancers who do not have a musical background, or for musicians who do not have a movement background, there is a context with which they are unfamiliar – hence the need for translation. The different contexts highlight the different languages (metaphorically speaking), per se, and so too the different ways of understanding. This issue is of central concern to this chapter as artists and audiences are able to understand in multiple ways, through their different experiential (translation) contexts. The arts are understood through and by language, and that language is specific to the art form, though equivalences exist: for example, dynamics exist in both music and dance, but in the former we refer to volume changes, whereas in the latter we refer to muscle intensity. As Jordan remarks, there is both a problem of language (2000) and therefore a sense that one needs to learn an “alien vocabulary” in order to engage in multimodal discussion (Jordan forthcoming). As most choreomusical scholars note, the different notational systems represent one key barrier to sharing the process of creation. Other dance scholars also remark on dance in such a way as to refer by analogy to language: that dance is a “culturally codified and meaningful human action” (Buckland 2007, 187). Beyond verbal and textual language though, we also have a somatic experience of live performance arts (akin to language issued via braille), and as such the arts are understood through and by our senses: like spoken discourse, we hear the music (or verbal language) and we see the dance (physical gesture). The audiovisual interconnections are mapped cognitively in our total experience of the work as we read, hear, and feel the gestures: those gestures are the moments that are understood as significant in our personal interpretative translation of the work (Minors 2012b; Zbikowski 2012a). As Peter Dayan remarks in his analysis of the inter-­art aesthetic of early twentieth-­century works: “There can be no direct translation and no unproblematic collaboration” (Dayan 2011, 3). The problem referred to by Jordan (in relation to language analogies) and Dayan (in relation to the transfer of sense across and between the arts) is in fact the kernel of creative exchange, as Eisenstein observed in relation to early film: “this relation is not one of similarity, but, as a rule, one of question and answer, affirmation and negation, appearance and essence” (in Albright 2004, 93). Jordan, in her recent work on choreomusicology, asserts a significant point which reminds us that we should look not only to difference but also to the shared points: Whilst we might still be able to trace the separate development of the two media, these two sensory planes now meet to affect each other and to create a new identity from their meeting. The old terms of congruence and non-­ congruence, similarity and difference, now seem inadequate. (Jordan forthcoming) Bringing this back to the field of multimodal translation, there has been recognition that language and music offer particular forms of understanding: “The battle

160  Helen Julia Minors

between ‘logocentrism’ and ‘musicocentrism’ has raged for centuries, and is far from over” (Apter and Herman 2016, 6). I would add here that the particular context and perspective from which the artist comes offers their own battle, which might in fact be “dancentrism,” where another way of experiencing and communicating is dominant, other than music and verbal-­textual language. As Susan Bassnett claimed, translation exists within a “multilingual and multicultural context” (2014, 1). Drawing on intercultural arts research might offer new insights into considering how and where music and dance meet, how they function together, what they produce, and how they produce what they do together as a single work, created from the catenation of different elements. In considering these modes of expression and communication, we ask how we are able to transfer sense across boundaries: they may exist within and across art, language, and cultural domains. The act of translation requires the translator to be creative. Venuti’s conclusion that “no act of interpretation can be definitive” asserts the need for creativity, emphasising the individuality of the translator (Venuti 1998, 46); in the context of choreomusical studies, I highlight that the translator is not invisible, but rather poignantly visible and audible. The arts are understood through and by culture. In drawing on translation here to reassess the interrelationship of music and dance, I, like Pamela Burnard et al. (2018), use translation to interrogate my field in order to ask what dance-­ music works are and how they express themselves. “[I]ntercultural translations are co-­constructed through collaboration” (Burnard et al. 2018, 232). The issue of collaboration is central to multimodal creativity and so it is central to multimodal translation. All music-­dance works discussed here are co-­created and all rely on an exchange of understanding in collaboration and performance to achieve the resulting work. The cultural context of any work and the performance of the creative act inform meaning construction. A particular identifiable movement, such as an arabesque, has particular cultural references. Notably, the French name of this dance movement means “in an Arabic style.” It is usually performed on point, with a raised leg, at 45 or 90 degrees, and is known as à demi hauteur or à la hauteur. The meaning of physical movement is reliant on its context. In music, an arabesque is likewise reflective of Arabic styles: the highly embellished style of Arabic architecture is reflected by highly embellished melodies. Whether meaning delivery has been issued by verbal, sonic, or physical gestures, each is created within a cultural context, or across cultural contexts. Similarly to Dayan, Gunther Kress, in discussing multimodal communication, asserts that: I assume that “translations” across modes within a culture are both possible and hugely difficult [but] that translations across cultures, whether the same mode ... or across different modes are also possible, though always achieved with enormously difficult selection; at a considerable level of generality; and inevitably with significant changes in meaning. (Kress 2010, 10)

Translations between music and dance  161

I identify therefore three key approaches to the translation of music and dance as communicative active media: senses, languages, and cultures. The problematic nature of artistic collaboration, language delivery, meaning transfer, and expressive exchange is necessary to all forms of multimodal artistic collaboration and in fact the reason why translation in this context deserves assessment. These three categories of meaning exchange were also identified as a result of the Arts and Humanities Research Council’s funded research project “Translating Music.”1 The difference in this chapter, however, is that we are considering the somatic experience of music-­ dance works and the sonic-­visual dimension of such multimodal works. Kress defines mode as: a socially shaped and culturally given semiotic resource for making meaning. Image, writing, layout, music, gesture, speech, moving image, soundtrack and 3D objects are examples of modes used in representation and communication. Phenomena and objects which are the product of social work have meaning in their cultural environments. (Kress 2010, 79) The list of modes here is interesting for this discussion, as dance crosses all of these modes. It exists in and through gesture and image, with and between musics, on screen, and as a multidimensional body, and it does this even if, as Matthew Reynolds suggests in his chapter for this volume (pp. 118–19), it is not always clear exactly where one mode ends and the next one begins. The capacity for dance to function across all of these modes asserts its role as a medium (as an artistic medium which functions within the collaborative whole, which expresses the design manifest in corporeal medium) to be used in this translation process (Kress and van Leeuwen 2001, 6). Likewise, music exists alongside speech and text, embedded with gesture, and functions on screen. In response to Kress then, I assert dance and music as modes that can be involved in translation: “What a community decides to regard and use as mode is mode” (Kress 2010, 87). The three categories (sense, language, culture) provide ways in which we might assess the translation process that takes place between music and dance. The notion of translation therefore invites us to explore the ways in which music and dance speak to each other, the ways they adopt each other’s processes or contrast the other in a way that formulates an active dialogue across the arts, within a specific performance space, for a specific performance event. In order to approach this interdisciplinary field, I draw then on a background in three areas: (i) music and dance studies, often referred to as choreomusicology (Minors 2006, 2009, 2012a, 2012b); (ii) music and translation studies (Minors 2013a, 2019); and (iii) intercultural arts research (Minors 2016). Drawing on these varied fields, I attempt to tackle the concern raised by Kress that “ ‘translations’ across modes” (2010, 10), across multiple cultures, might dilute the observations. Speaking from within multiple fields (though this is not exhaustive) enables me to suggest three categories to the translation of music and dance, which do exist between

162  Helen Julia Minors

similarity and difference, and to speak from the point of liveness – the lived experience of the work whether from the perspective of the creative collaborative artist or from the point of felt, lived experience of interpreting the work in performance. Choreomusical studies would benefit from exploring the concept of translation, in order to further assess how meaning is transferred between these artistic media. This approach looks at translation as a way to understand music-­dance work. It seeks also to understand the dialogue and interaction between music-­dance and to explore how considering the three categories (senses, languages, cultures) might offer new insights for choreomusical studies. The chapter explores first translation in the context of choreomusical studies, before using three case studies to illustrate how translation can offer new perspectives to interpreting music-­dance works. How might translation facilitate new understanding of the creative and receptive dimensions of music-­dance works? Can it reveal something of the creative choices made by composers and choreographers working in collaboration?

2. New approaches to music-­dance translation For choreomusicology, the temporal nature of these movement arts, which are discreet due to their audiovisual nature, requires us to theorise the audiovisual transfer, but in order to do so acknowledgement of the problems of language must be raised again. As Şebnem Susam-Saraeva (2008, 189) notes in her seminal study on music and translation: “The mere mention of translation within the context of music opens a huge can of worms for many researchers and practitioners.” The can is expanded when we consider dance as well as music. Translation is of relevance to many musical texts (Minors 2013a), and in asking how “music speaks” (Albright 2009), it is necessary to ask who it speaks to, how it speaks, and what it says to dance and music in particular. Despite the difficulties in translating across media, the examples in this chapter have a shared common practice in that I select balletic examples in which there is a culture of music-­dance works produced for the theatre. As such, the domains of music and dance are shaped by their location within the genre of ballet, and more specifically within a Western twentieth-­century balletic practice. Collaborative music-­dance works are complex due to the fundamental differences between how we produce and experience these arts. There is a sensory divide, but it is not an impenetrable wall. We assume difference due partly to the name we give these arts, but some cultures have one word to embrace them both. Theorists recognise the overlap of the many arts: Daniel Albright’s pivotal work on panaesthetics proposes that the arts are not separate and that in fact “All art is inscribed on the body” (2014, 281). Moreover, in establishing The Principles of Art (1923), Robin Collingwood claims that “the dance is the mother of all languages” (in Copeland 1983, 371). The reciprocal nature of these arts is highlighted when Albright notes that “A dance is a response to music, and a making of its own music” (2014, 284). In essence, Albright asserts the well-­cited remark of Walter Pater that “all art aspires toward the condition of music” (Herzog 1996, 122). Whatever the condition (cultural context, performance location, or message), the arts can function in the terms

Translations between music and dance  163

of the other, adopting processes and equivalences in order to exchange gestures. Although the arts form an interdependent relationship and can transfer content, style, and gestures, they cannot “be physically shifted, translated into the medium of another” (Dayan 2011, 21). The body has however always moved and predates language and formal semantic structures: in many ways the audiovisual cognitive mapping we enact when experiencing multimodal works could be said to predate and so underpin language, and therefore be central to all forms of translation studies. It is necessary to question: how do the aural and visual dimensions react to one another? Translation offers a way to look at not only an exchange of content but also the ways in which artists respond to each other by the choices they make.What choices are made by composers writing for dance, and particularly for ballet during the twentieth century and beyond? What choices are made by choreographers in setting dance to music (whether they chose pre-­composed music or work in real-­time collaboration)? Indeed, how does an act of collaboration between music and dance rely on a process of translation? Both music and dance have their own vocabulary and syntax (metaphorically speaking) in the ways in which they put their art together: it is a sense of musical metre, or a sense of dance metre (as one example), which is different from verbal language. Henrietta Bannerman observes that this language is “meta-­kinetic rather than metalinguistic” (2014, 66). The difference between textual language and music-­dance language is grounded in the experience of movement and non-­semantic musical sound. This is not to say that music cannot be referential in its meaning, but it cannot represent specific words or the nuances of sentences. Fundamentally, much of the equivalence is drawn out in the experience of time, and therefore of meter, rhythm, phrasing, and tempo. Reacting to the cultural turn in audiovisual translation (Apter and Herman 2016) and to the continued concern with cultural musicology, this chapter requires this theoretical overview of translation in this particular music-­dance context. The concept of musical gesture (Minors 2012b) as an analytical approach to exploring how music and dance form meaningful relationships is central. Gestural analysis is proposed within the realm of conceptual mapping, where the integration and blending of domains is essential. We are aware of the separate art forms, their content and structure, but we experience the work as a whole, during the moment. As Massimiliano Locanto (2018, 38) remarked, in response to my previous work on music-­dance gesture (Minors 2012b), the many networks form from the different conceptual domains, and the mapping of those domains form a new perspective of the interplay and communication between music and dance. Developing a model to start to explore translation across music and dance, I expand a model I proposed elsewhere (Minors 2012b, 171), in which music and dance spaces are shown in a Venn diagram, to illustrate the blended space of the two domains. This model illustrates three categories through which the mapping of music and dance can be explored in terms of translation (senses, languages, and cultures) (see Figure 7.1). The mapping might be produced by the creative artists during collaboration and by theorists and by spectators in receiving the work. The translation perspective (be that an exploration of language, sense, or culture)

164  Helen Julia Minors

FIGURE 7.1  Mapping

music-­dance interdependency read through the translation of languages, senses, and cultures

feeds how we read the mapping of music and dance. It informs the ways in which we might interrogate those relations, ensuring that we go beyond the traditional notions of similarity and difference (as outlined earlier). The experience is situated within a particular context (era, venue, country). Notably though, if translation enables the creative artists and theorists to interrogate the relationships of music and dance as interdependent, then the model needs to recognise that there is a continual reciprocal exchange and transfer of content during the creative process and during the performance of the work, in which each art modifies the other. What we hear is modified by what we see, and so, what we see changes how we hear the music. In the field of intercultural research, similar questions are now being raised as well: Burnard et al. asks how both music and musicking might “function as technologies of translation that allow us to understand and conceptualize practices of diverse musical creativities” (2018, 233). This approach considers thus the participation of all involved in the works’ creation and reception. It extends to consider then, beyond textual-­verbal language, how music is translated and how much it can translate; how dance is translated and how dance can translate.

3. Translation and the gestural arts of music and dance Why is the notion of translation significant to music-­dance works and to the growing field of choreomusicology in particular? Theorising translation in this context

Translations between music and dance  165

aims to illustrate its significance as a new approach to choreomusicology and to refine the fact that music translation is not only the translation of various musical texts (such as programme notes, lyrics, or transcriptions), but it also claims the capacity for dance and music themselves to translate the content of other artistic media. Through the following three case studies, I ask: can a process of translation be read when choreographers work with pre-­composed music? Or on the contrary, how do composers and choreographers utilise a process of translation and transfer when they collaborate in forming new music-­dance works? Considering the creative process, we then turn to the resulting work to ask: how do we read those emergent meanings between the audiovisual elements? A caveat is required: translation as a term is in danger of becoming overused and might lose its usefulness unless it is refined. The term in music overlaps with notions of meaning transfer, artistic adaptation, arrangement, covers, rewriting, transcriptions, and transformations (Minors 2013a). As a process, translation enables researchers to question the choices of the creative collaborators, to chart the developments in the field, and to find ways to analyse audiovisual works. As such, it is interrogative in nature. It requires the mapping of domains. But it does not exchange one note for a movement (or a word for another): there are no claims for the exactitude of meaning or syntax. What are the implications of translation for music-­dance works? If a choreographer, for example, tries to translate the content of a musical score into dance, might the result only mimic music through elemental equivalence (rhythmic mimicry, for example)? Or, do those equivalences, which are read between the audio and visual domains, produce a greater variety of interrelationships which reveal a state of interdependence (namely a state of co-­existence where each relies on the other, but each does retain its own autonomy within the whole ephemeral experience)? Notably, I focus on narrative balletic works to refine the scope of the example. I utilise three case studies to illustrate the three approaches to the translation of music and dance outlined earlier: (i) of a composer and choreographer collaborating together, as well as an example of the same choreographer selecting a pre-­composed score, to illustrate the translation of sense across the audiovisual divide (sensory translation), which relates to Kress’s notion of “moving meaning” in translation (2010, 124); (ii) of the use of music visualisation, to unpack how the language translation of music and dance can be understood as translating one another, as an intersemiotic translation seeking language translation by analogy, ultimately seeking to visualise the music, or to audioise the dance, which relates to the subtype of translation, transduction, denoted by Kress, whereby “meaning-­material [is moved] from one mode to another” (2010, 125); and (iii) of an (inter)cultural translation of a score interpreted twice, a century apart, in order to show the re-­appropriation of music and dance into a new cultural context, whereby new meanings are imposed but some meanings are also carried forward. As stated earlier, a cross-­reference to Kress is possible, as the “re-­ ordering of the elements” occurs “across cultures in the same mode” and so represents a subtype of translation he labels “transformation” (2010, 129). The examples I have chosen to draw on are specific to the need for further research and discussion across music and dance: the initial two case studies are

166  Helen Julia Minors

chosen for their popularity in the field of choreomusical studies and the quality of the music and choreography. By taking examples that have been previously researched and analysed, I aim to bring the new lens of translation to them to argue that the process of interpretative translation is one that can facilitate better understanding and engagement across the disciplines. The final case study is chosen to acknowledge that there are meanings beyond the collaboration which emerge to engage with contemporaneous culture, politics, and society. The example illustrates a re-­interpretation and modernised adaptation which is grounded very much in cultural history and not only lends itself well to the cultural turn in dance and music research, but enables me to iterate the importance of considering translation as an appropriate and engaging tool by and through which to explore music-­dance works. A deliberate choice has been made, therefore, to select case studies which are already well trodden from a variety of perspectives, to add the concept of translation to these seminal works and to offer a new perspective to choreomusical studies.

4. Sensory translation as an approach As mentioned briefly in the Introduction, Vaslav Nijinsky was a lead dancer of the famous Ballets Russes, led by Serge Diaghilev, and he was commissioned by Diaghilev to create new choreography for two works: the first, Claude Debussy’s already successful Prélude à l’après-­midi d’un faune, for their 1912 season, and the second work, Igor Stravinsky’s Le sacre du printemps, for their 1913 one. Debussy’s work had been composed in 1894 in response to Stéphane Mallarmé’s symbolist poem of the same name, while Stravinsky’s work was composed in response to his own vision of a sacrificial rite in which he saw a young virgin danced herself to death. Debussy’s choice is particular: the music, like the poem, is arguably symbolist, offering 110 bars to respond to the 110 lines of the prose poem (Wenk 1976, 161); it also offers the repeated melody once on a different tone centre of A (bar 23), as the poem likewise refers to tuning to the A (line 37). There is direct music-­ text correlation. The symbol of the panpipes is shown through the flute solo (for example, at bars 94–99), and as two pipes are referred to (line 20); the music also uses two unison flutes earlier in the piece (which has no significant musical impact whatsoever, so seems certain to respond to the text, moving from dividing the motive between them to playing together from the climax of the phrase, bars 26–29). Likewise, the sexual chase is illustrated through the imitating horns (known for the illustration in nineteenth-­century opera of the hunt). Importantly though is the sense of movement felt in distinctive ways through and between music and dance. This sense of movement in the moment is integral to a sensory translation, as this relies on a somatic experience and an experience of the ephemeral qualities of music. I claim this example as one of sensory translation between music and dance (and one could argue by extension to the ekphratic source of Debussy’s music, the poem) because the shared response to the poem and the shared emergent meanings are developed through different artistic processes, but the end sharing of sense

Translations between music and dance  167

is coherent. The blend of the senses is significant. The poem places the idea of dreams in opposition to reality, and the solo faun who lusts after the mythical nymphs seems to be across both states of existence. The music conjures up this variable state of existence through two key means: (i) the solo melody avoids all forms of conventional tonal harmony, avoiding establishing the key and avoiding any closed cadences – the music seems suspended around the tone centre (C#); and (ii) the opening statement (flute solo, bars 1–4) is repeated many times, with five statements in the first section (bars 1–29) – Section A1 uses modified material but includes the opening modified statement twice before further developing material (bars 31–36). A nocturne section follows. The reprise of the original statement occurs at bars 79–82, before four further reprisal statements. The return of material could be understood as memory. Importantly though, the changes to rhythm, phrasing, and to the length of the statements modifies the experience of time. Debussy creates a statement which is languid and does not have regular beats, but pitches duple and triple patterns side by side within a phrase, offering an improvisatory feel to the statement. The sustained opening note is held and sustained during its third statement (bar 21) and following. The reprise statement uses a new time signature, focusing on duple meter and regular beat patterns (bars 79–82, see Figure 7.2). Reprise statement 3 is similar to the opening but now in 4/4 rather than 9/8 time (bars 94–99) (see Figure 7.3a bar 1 and Figure 7.3b bar 94). This is significant as Debussy makes great efforts to distort our experience of time, but to include constant musical motion. Nijinsky’s response to both Mallarmé’s poem and Debussy’s music ensures to retain the uncertain state

FIGURE 7.2 

Bars 79–82

FIGURE 7.3A 

Bar 1

FIGURE 7.3B 

Bar 94

168  Helen Julia Minors

of whether the faun (of the title) experiences the situation, or whether in fact he dreams it. This deliberate doubtful state of existence is advanced in how Nijinsky choreographs movement to Debussy’s score. Whereas Debussy uses constant rhythmic movement with changeable duple-­triple rhythms, Nijinsky offers a static form of dance whereby the faun poses, lying on his side atop a mound, for the opening statement. The physical stasis gives a sense of motionless to the music and in fact enables a listener to draw out the similarity of the music, not in stasis, but in the repeated aspects of the opening statement (the descent and rise to and from the same note, the exact repetition of bar 1 immediately in bar 2) (Nijinsky 2013, time code 00:00–02:07). The fact that the faun also holds a pipe and mimes playing it unifies the connections between the audiovisual domains and tricks the listener into imagining the faun playing this melody; as such the theme becomes associated with the faun and its reprisal therefore with his changing states of being. The equivalence of music and dance languages is considered in order to use their difference to project a particular sensory experience of motionless and a dream-­like state. The audience is invited to “acknowledge the autonomy of each” (Duerden 2007, 77) art form due to the different sensory mode of delivery, but this does not result in difference only. The contrasting artistic features produce a consistent emergent meaning through their mapping. The difference in process and production are able to result in similarity of meaning. Let’s test this further: the pedestrian walking movements of the nymphs, moving in a two-­ dimensional-­like side-­on presentation, much like Egyptian hieroglyphics, adds to the sense of stasis, notably at their presentation with the fifth statement of the theme (bar 26–30; Nijinsky 2013, 02:53–03:30). The lead nymph slowly holds out her scarf and places it on the floor, further adding to the concrete motionlessness via the object being relinquished. Notable too is Nijinsky’s response to the repeated flute turn motive (bar 27) in the angular, stuttering movements of the faun’s head. The minute physical movement accompanies a complex alternation of the two flutes: his contrasting movements give a further sense of smoothness to the musical line. Nonetheless, again the audiovisual difference comes together, meeting in a blended space, to offer a sense of unity where the faun and nymphs lay eyes on each other. The uneasy transfer between the audiovisual elements is necessary to convey the dream-­reality binary. That dance has a “refusal to cooperate” (Albright 2000, 7) with the music, and this is central to using the different elemental features of these art forms to combine to issue a single narrative. Furthermore, there is also an analogy between the artistic media of music and poetry: dance offers unity of message at the first point where two flutes play in union. The levels of complexity one can read when considering how dance might translate the sensory experience of the music affords the fact that equivalence of unity may be used, despite the difference of rhythmic movement. Only together do music and dance formulate the resulting audiovisual image. The notion of mutual implication proposed by Claudia Gorbman (1980, 189), though rather old, stands fast in research which explores the transfer of sense and the exchange of ideas across audiovisual media.Whereas traditional notions of translation

Translations between music and dance  169

rely on words through speech and text, artistic translation relies in part on the sensory experience, various ways of “speaking” or issuing content, and a variety of ways of documenting that content (for example, the score, audio recording, dance notation, video recording). As such, Michel Chion’s approach to film theory, noting that a “transsensorial perception” is necessary (Chion 1990, 137), is highly pertinent to this current context, too. The constant exchange between media enables the further emphasis of the dream versus reality as the spectator is drawn to question from whose perspective we are viewing this activity. Even the faun asks:“Aimai-­je un rêve?” [“Was it a dream I loved?”] (Mallarmé, in Austin 1970, 23, line 5). Further emphasis that a sensory translation might offer more to the work is shown in Mallarmé’s own response to Debussy’s music, when he claimed that the music “prolongs the emotion of my poem, and sets its score more vividly than color” (Austin 1970, 13). Nijinsky’s approach to working with a composer was of course different: in working with Igor Stravinsky on Le sacre du printemps, he worked with creating new metrical structures and a very different form of choreography again, which disregarded conventional balletic gestures, preferring instead knocked knees, flat feet, angular movements, and a distortion of the body (Nijinsky 1987). Stravinsky’s score was created to his own vision of a sacrificial rite where a virgin would dance herself to death (a tale often repeated by Stravinsky in interviews and documentaries). Nijinsky seems to have felt the multiple characters of the scene, in which young girls, sages, and senior figures compete in their role in the ritual. Responding to the friction in the story, and to the many musical textural layers, we see for the first time a curtain lift on the second statement of an extremely high bassoon solo (which is based on Lithuanian folk music, Stravinsky 1947, rehearsal figure 12), not a dance soloist, duet, or corps de ballet, but on a scene where there are distinct groups, with a circle of men back left, an old woman alone and knock-­kneed and bent over at the front, with a circle of girls kneeling over at the centre and another group at the back right, with a further group crouching at the front right (Stravinsky 2008, 02:38). Stravinsky’s music is grounded in metrical structures, which use regular rhythms off set with syncopation. Here the 2/4 metre offers four regular quaver beats in a bar, but the accents disrupt the flow. Nijinsky, seeking no doubt to enforce the primal sense of the story and to emphasise the primitive nature of this rhythmical accented music, matches the arm movements of the first male group to the accents of the music: arms rise and fall only on the accents. The feet all the while move with the four quavers as though the accent were on the first beat of the bar. The choreographic approach, in matching physical movement to musical movement, is in complete opposition to his approach to choreographing Debussy’s music. The physical matching of the different rhythmic layers of the score has the effect of intensifying how the accents are felt and seen. Both seem stronger together than alone. Interestingly, “Stravinsky conceives of language as normally not only expressive, but invasive, claiming the right to translate into its own logic the functioning of all media, including music” (Dayan 2011, 125). Stravinsky spoke of music-­dance relations: what is striking is the importance for him of the rhythmic correspondence, as he refers to the correspondence being “measured to the musical unit” (Dayan 2011, 17). This level of invasiveness seems to impact on

170  Helen Julia Minors

the collaborative relationship when we look to the rhythmical structures used in Nijinsky’s choreography. Both of these examples illustrate some of the early seminal changes in twentieth-­ century narrative ballet and in musical composition, as the old traditional notions of form, metre, and harmony were being challenged. The sensory translation propagates that challenge to their own disciplines while also superimposing them for the spectator. The examples also illustrate how a choreographer, who is new to his role, responds to and translates these composers’ revolutionary approaches to texture, timbre, rhythm, and thematic development.

5. Language translation as analogy Language translation is most keenly illustrated through examples of music visualisation, where the elemental features and syntax of music and dance are read by the choreographer as analogous to language. The elemental features of music, such as rhythm and pitch, are read as equivalences to dance rhythm and height (the metaphorical equivalent of pitch). The approach to translate the content of one art form similarly into another has caused some concern over mickey-­mousing, a state whereby the different domains aim to mimic one another. Music visualisation requires “analogical thinking,” whereby there is direct correlation between the elements of each domain (Zbikowski 2018, 60). As Paul Hodgins identified when he established the term choreomusical, “the degree of music visualization in dance depends to a large extent upon both the choreographer’s own predilections and the translatability into movement gesture of the musical topology involved” (1992, 13). This process starts with identifying similarity in terms of those elemental features, but it develops to embrace complexity as we map the domains to form structural correlations offering more than the sum of their parts. Carroll and Moore remarked that “the dance movement functions as a translation from one medium to another – from the musical movement impulse into flesh and blood movement” (2008, 16). They claim that the most intriguing music-­ dance relations are those whereby there is “embodied translation of the musical motion impulse” (Carroll and Moore 2008, 17). An early proponent of music visualisation, whereby music is the catalyst for dance, was Doris Humphrey. Her Air for a G String (1934) shows that pedestrian movements (walking, steps, waves) move with the same rhythmic motion as the music. The lower levels of the body correlate to the bass line of the music while the hands and upper body correlate to the melody (Humphrey 1934, 00:00–00:26). Humphrey was very aware of the audiovisual processing of the audience and the role of translating the audiovisual experience. She warned that “not only is the eye faster, but, in a contest with the ear, will invariably take precedence” (Jordan 2012, 225). Her awareness, also relevant in much film theory, draws out the fact that in mapping the audiovisual components, there is not always equality. The speed with which we process these audiovisual arts is different; the senses by which we digest them are also different, and our experience of “seeing” a production dominates the eye. Interestingly then, that many choreographers

Translations between music and dance  171

have sought to work with this notion by visualising the music. Do we therefore hear the music better when we see their dance? How music impacts dance is so significant to Humphrey that an approach considering translation explores not only that there is an effect, but how the effect has resulted from the creative process. The impact of music is so great on dance in this context that it changes the way it is seen and understood. Rather than experience the individual arts, the audience is now invited to “one whole experience, music and dance inextricably combined” (Jordan 2012, 226). As commented later in the twentieth century, the famous choreographer George Balanchine remarked that “music must be seen” (Zbikowski 2012a, 219). But how is it seen here? What exactly is seen, and how does it project an analogy? The elemental features are considered as equivalents often as the nouns which label them are the same between the arts. But fundamentally the importance resides first in the co-­existence of these as temporal and therefore rhythmic arts, able to move in time and to shape, divide, and structure time. Humphrey responds to Bach’s Air on a G String by illustrating the musical phrases in the movement lines she depicts, as well as ensuring, as noted, the rhythmic structures are seen. Ultimately the transfer of process, of music visualisation in dance, is not an end in itself. It is one way to bring a new work to life, but it requires a challenge – that instability in translation affords a benefit to such a creative situation. The choreographer must add something and must preserve qualities of both the arts. Their autonomy and their relationship is important, hence the notion of interdependence: the combination must be greater than the sum of its parts, but it must also respect the distinctive features of each domain. The process of mapping the analogy does not remove the significant agency of each domain. The most current and notable example of music visualisation comes from the choreography of Mark Morris. He often uses pre-­composed music and works from the musical score. In using Purcell’s Dido and Aeneas, he created a new work, which incorporated a live soprano on stage, ensuring the possible gestural communication in the moment between musicians and dancers. His Dido’s Lament (1995) illustrates how choreographers use music visualisation to translate the sonic musical elements in movement terms and in particular to the structure (the passacaglia) and to the lyrics (physical and sonic gestures metaphorically convey the meaning of the libretto). It points also to the importance of differences that result in gestural content between music and dance, as the visualisation is not simply mimetic. Morris’s musical understanding is particular among contemporary choreographers, and he is rare in also conducting his works. Significantly here though,“Morris would refer in interview to his musical approach invariably in terms of translation” (Jordan 2015, 77). It seems this approach is unique. It highlights the process of his creativity; it gives voice to the translator (rather than making the translator invisible). Morris has a particular role in creating his dances and then translating the content onto bodies, into performances, and considering the interrelationships of the arts as he acts as choreographer, dancer, and conductor. Moreover, many of his dances were made on himself, using his own body to choreograph the movement

172  Helen Julia Minors

on and then to perform the work, including the premiere of Dido’s Lament.Various processes of translation are enacted then when he interprets the musical structure, the visual realisation, the embodied experience. His approach is embodied, but as he draws from the music as a starting point, we must ask whether his “dance offer[s] us new insights into music?” (Jordan 2018, 88). Certainly, ground bass (repeated bass phrases) and canons (imitative melodic structures) are visualised in such a way that a listener may become aware of musical repetitions which would otherwise become part of the larger musical texture. The opening ground bass here (Morris n.d., 02:17 onwards) sees the bass rhythm represented in the feet of the corps de ballet. As the bass line descends, the dancers move upstage (towards the rear). As others have noted (Jordan 2015), Morris used gestures which are analogous directly to language in embedding sign language, notably for “memory” (04:06) and “breast” (03:33). These symbolic gestures also assert and bear out clearly his approach to interrogating the analogies of the temporal arts. The combination of music and dance is not one of “marriage,” as Damsholt (2002, 238) remarks in relation to our metaphorical understanding of music and dance through binaries. It is less consensual in that, to borrow from Jordan, “music infects the dance” (2012, 226) in this particular choreographic process. The journey to seek analogies and to visualise music places importance on the sonic elements in such a way that music can “impose ... its own characteristics on the other” artistic media (Cook 1998, 103). There is a deliberate choice though with these illustrated choreographers. Music as source text does not mean that dance, the target text, only represents music. The reciprocal nature whereby the analogous experience of these temporal arts is mapped means that both change our perspective of the other, as a spectator. This identifies that any translation adapts the original source text and that the target text, although containing content of the source text, is never identical, and neither is that the aim in creative works. The interdependency of music-­dance works extends from this necessary reciprocal exchange (even when this exchange draws out from language analogy). If one were to perform from music a “scientific translation into bodily action” as St Denis remarked in discussions of music visualisation, the focus must reside on those elements where equivalences can be found, including “the rhythmic, melodic and harmonic structure” (in Damsholt 2018, 22). This formal plotting of musical gesture to danced gesture is not a strict translation and it privileges a limited number of elements (especially rhythm). It reminds us that the choreographer in these instances has many choices to make. It is a process whereby some of the content of one medium, selected by the choreographer, is transposed to another media. As such, the meaning and style of the work changes – exactitude is not carried forward.

6. Intercultural translation as commentary Cultural translation can be seen in many areas, especially in music, where styles and genres migrate around the globe, modifying according to cultural location and era. Music is rooted in diverse cultures across all races. As such, the choice to illustrate

Translations between music and dance  173

cultural translation is selective but remains in line with the previous twentieth-­ century balletic examples. As “culture filter[s] our experiences of the world” (Sturge 2011, 67), it is imperative to consider the cultural contexts of music-­dance works. There was a recent revival of Erik Satie’s La Parade (originally choreographed by Leonide Massine, with a libretto by Jean Cocteau, designs by Pablo Picasso, staged by Serge Diaghilev’s Ballet Russes in 1917 [Satie 2007]) produced by National Dance Company Wales (Satie 2017) to celebrate the centenary of the Russian Revolution. But, how can a music-­dance work examine cultural translation? What is done to the work to revive it? In the new version, P.A.R.A.D.E. (carefully re-­spelt in capitals with full stops between each letter), there is a clear cultural and temporal transplantation of a French work, performed now in Wales, to commemorate the Russian Revolution. As such, the work (the score and aspects of the narrative) are resituated into a new location (different culture, era, modified narrative) with new emergent meanings. The new version has new choreography and modified scenario, for a new translation of the music a century after its original premiere. The BBC broadcast of the work describes it as a “bold new take” on Satie’s score (P.A.R.A.D.E, 00:10–00.30). The director, Marc Rees, utilises the conflict present in the political and technological changes which have encompassed the globe in the last century. This is notable as Satie’s La Parade was set to Cocteau’s play in which economic business was subverted through the lens of the circus managers and a marketing ploy to sell tickets to the next show. The original was a stark satirical commentary on globalisation and technological advancement with costume and set designs by the cubist artist Picasso: amongst the designs are managers’ costumes constructed from skyscrapers and horns, as well as a knock-­kneed American girl and a Chinese Conjurer. The cultures represented were explored through stereotypes and visual cultural symbols, but the musical score was not immune to such subversion. The siren, lottery wheel, typewriter, and so on embed the sounds of the city. As such, this score and narrative is pregnant for a revival. The new setting likewise has a political dimension, an economic dimension, and a technological dimension. The translation might move the work to a new narrative and new context, but some of the context remains and is mapped to the new context. The circus is reimaged as a factory. The technological advancement of skyscrapers is now transplanted to a humanoid robot. The outside location of the managers selling tickets for a circus is now an immersive event where the audience becomes the protestors against a political speaker. Armed with a flag, on which a capital P. was written on a red background, as an audience member I was encouraged to shout against the political speaker and the robot, and to march from the outside into the factory (the Millennium Concert Hall, Cardiff) to experience the performance as though the workers had prepared something to display to the senior political figure. The notion of outside and inside, us and them, is a stable feature across both the 1917 and 2017 versions. The technological ideas in Satie’s score, not only through the typewriters et al. but also through the repeated ostinato and mechanistic rhythmic passages which return throughout, and the angular accented melodic lines, were used in movement

174  Helen Julia Minors

by correlating repeated angular movements. In movement, however, the mechanisation is shown through a wall of cardboard boxes which is slowly dismantled (the curtain rises to show the wall, 09:56). The impenetrable wall of translation referred to earlier is made live here and is disrupted, broken down. The first dancer to appear is folded into a box and slowly works the contorted body out of the box, leaving feet contained and held, restricted. The collaborative team was vast: National Dance Company Wales, led by director Marc Rees, were joined by BBC National Orchestra of Wales, as well as Rubicon Dance and Dawns i Bawb. Choreographer Marcos Morau was joined by graffiti artist Pure Evil, architectural designer Jenny Hall, and aerialist Kate Lawrence in order to develop the introductory immersive event. Using segments from Satie’s introduction to Parade, musical motives are diffused through speakers in Cardiff Bay, disrupted by static and the sounds of the protesters (04:35–04:45). The performance boundaries are dissolved as audience members become participants. Though as noted earlier, the role of translation can be invasive where one media imposes its characteristics on another. Here this imposition is on the audience and local community, with tall high-­r ise flats surrounding the commotion hearing and seeing the activity from their balconies. The ballet production is subversive – the robot is subverted to rebel against the political leader, working with and for the workers – much as Cocteau’s original scenario had integrated difference and diversity. Like the surreal nature of the first performance, the choreographer utilises the objects of the everyday working environment to impose on the dancer – tape is used not only to seal boxes but also restricts the dancers, as they wrap it around themselves and others and across the floor. The ostinato and rhythmic repetition is seen in the danced movements, although they are deliberately not synchronised, to convey the rebellion taking place (focusing on the cultural meaning of the work, avoiding music visualisation). Finally, the robot appears at the end alongside the workers, but notably the workers, having previously de-­robed down to white underwear, now wear costumes which they have made live, on-­stage, during the performance, from the cardboard boxes and tape: they become mechanised themselves. To ask where dance and music meet, one also needs to ask how they come together, and in coming together what they do. In other words, what does their joint, collaborative activity produce and how do they produce it? Bannerman noted that “from the point of view of structure, dance shares commonalities with language and [that] like language it communicates according to cultural codes” (2014, 66). These cultural codes are shared here through the meta-­themes of technology, economics, and political change. Not only has 1917 been brought forward to 2017, in recognition of the Russian Revolution, but the commentary, political speech, and audience inclusion acted further as a commentary on UK politics, with direct references made to the 2017 Conservative Party conference (with coughing interrupting the political speech). The immediate moment was a distillation of a number of eras, events, locations, cultures, and political responses. The mapping was complex but generated from shared cultural experience. This shared experience facilitated the translation both of the co-­creators and the spectators.

Translations between music and dance  175

7. Conclusion This chapter aimed to explore how music and dance work together, using translation to begin to introduce how translation is relevant to the creative and receptive dimension of these multimodal arts. In questioning how sense is transferred between the arts of music and dance, the three case studies reveal emergent meanings (Cook 2001) in the interplay of these temporal arts read through three potential categories of translation: senses, languages, cultures. The range of examples is unified by the choice of narrative ballets drawn from the twentieth-­century onwards. The three themes of translation in music-­dance works, which are those relating to language translation, sensory translation, and cultural translation, show that translation enables scholars, as well as creative artists, to interrogate what they do and how they do it, but primarily it recognises a process of translation as a fundamental act within all creative collaborative works. When referring to narrative film music, Gorbman asserted that “The point is that image, sound effects, dialogue, and music-­track are absolutely inseparable during the viewing experience, and they form a combinatoire of expression” (1980, 190). Translation is a bringing together, not a separation. The three case studies show that the notion of interdependence is vital to understanding these works as narrative ballets, but also to understanding their artistic gestures. Interdependence is created as these arts and artists are in dialogue – and dialogue cannot exist without translation of some form. Collaboration and creative work rely on the fact that the notion of translation is not precise. “Translation in art must never be secure” (Dayan 2011, 133). As such, in asserting a new cultural location (cultural translation), or in mapping music and dance which respond differently through different elements to the same stimulus (sensory translation), or in mapping the dance visualisation of music (language translation), the mutability of music and dance is co-­creative and is further interrogated in performance through the concept of translation which seeks not only exactitude of message, but endeavours to look to the transference of process as well as message.

Note 1 “Translating Music” (online) was a Network funded by the Arts and Humanities Research Council, which I co-­led with Lucile Desblache. Each of these three categories of translation (Sensory, Language, Cultural Translation) has been outlined in relation to opera, music accessibility, and education (Desblache and Minors 2019).

References Albright, Daniel. 2000. Untwisting the Serpent: Modernism in Music, Literature, and Other Arts. Chicago: University of Chicago Press. ———, ed. 2004. Modernism and Music: An Anthology of Sources. Chicago: University of Chicago Press.

176  Helen Julia Minors

———. 2009. Music Speaks: On the Language of Opera, Dance, and Song. Rochester: University of Rochester. ———. 2014. Panaesthetics: On the Unity and Diversity of the Arts. New Haven: Yale University Press. Apter, Ronnie, and Mark Herman. 2016. Translation for Singing. London: Bloomsbury. Austin, William W., ed. 1970. Debussy, Prelude to “The Afternoon of a Faun”: An Authoritative Score. London: W. W. Norton and Company. Bannerman, Henrietta. 2014. “Is Dance a Language? Movement, Meaning and Communication.” Dance Research 32 (1): 65–80. Bassnett, Susan. 2014. Translation Studies, 4th ed. London and New York: Routledge. Buckland,Teresa. 2007. “In Search of Structural Geist: Dance as Regional and National Identity.” In Dance Structures: Perspectives on the Analysis of Human Movement, edited by Adrienne L. Kaeppler and Elsie I. Dunn, 187–234. Budapest: Akadémiai Kiadó. Burnard, Pamela,Valerie Ross, Laira Hassler, and Lis Murphy. 2018. “Translating Intercultural Creativities in Community Music.” In The Oxford Handbook of Community Music, edited by Brydie-­Leigh Bartleet and Lee Higgins, 229–42. Oxford: Oxford University Press. Carroll, Noel, and Margaret Moore. 2008. “Feeling Movement: Music and Dance.” Review Internationale de Philosophie 4 (246): 413–35. Chion, Michel. 1990. Audio-­Vision: Sound on Screen. Translated by Claudia Gorbman. New York: Columbia University Press. Cook, Nicholas. 1998. Analysing Musical Multimedia. Oxford: Oxford University Press. ———. 2001. “Theorizing Musical Meaning.” Music Theory Spectrum 23 (2): 170–95. ———. 2013. “Foreword.” In The Psychology of Music in Multimedia, edited by Siu-­Lan Tan, Annabel J. Cohen, Scott D. Lipscombe, and Roger A. Kendall. Oxford: Oxford University Press. Copeland, Roger, and Cohen Marshall. 1983. What Is Dance? Oxford and New York: Oxford University Press. Damsholt, Inger. 2002. “The Marriage of Music and Dance.” In Of Another World: Dancing Between Dream and Reality, edited by Monna Dithmer, 237–49. Copenhagen: Museum Tusculanum Press. ———. 2018. “Identifying Choreomusical Research.” In Music-­Dance: Sound and Motion in Contemporary Discourse, edited by Patrizia Veroli and Gianfranco Vinay, 19–34. London and New York: Routledge. Dayan, Peter. 2011. Art as Music, Music as Poetry, Poetry as Art, From Whistler to Stravinsky and Beyond. Farnham am Main: Ashgate. Desblache, Lucile, and Helen Julia Minors. 2019. “Translating Music.” In Translating Cultures Encyclopedia. London: Arts and Humanities Research Council. Duerden, Rachel. 2007. “Dancing in the Imagined Space of Music.” Dance Research: The Journal of the Society for Dance Research 25 (1): 73–83. Gorbman, Claudia. 1980. “Narrative Film Music.” Yale French Studies 60: 183–203. Herzog, Patricia. 1996. “The Condition to Which All Art Aspires: Reflections on Pater on Music.” The British Journal of Aesthetics 36 (2): 122–34. Hodgins, Paul. 1992. Relationships Between Score and Choreography in Twentieth-­Century Dance: Music, Movement and Metaphor. Lewiston: Edwin Mellen Press. Humphrey, Doris. 1934. Air for a G String. Accessed December 10, 2018. www.youtube. com/watch?v=NjwJyaSIRqY. Jordan, Stephanie. 2000. Moving Music: Dialogues with Music in Twentieth-­Century Ballet. London: Dance Books.

Translations between music and dance  177

———. 2012. “Mark Morris Marks Music, or: What Did He Make of Bach’s Italian Concerto?” In Bewegungen zwischen Hören und Sehen: Denkbewegungen über Bewegungskünste, edited by Stephanie Schroedter, 219–36. Würzburg: Königshausen & Neumann. ———. 2015. Mark Morris: Musician-­Choreographer. London: Dance Books. ———. 2018. “Acts of Transformation: Strategies for Choreographic Intervention in Mark Morris’s Settings of Existing Music.” In Music-­Dance: Sound and Motion in Contemporary Discourse, edited by Patrizia Veroli and Gianfranco Vinay, 76–90. London and New York: Routledge. ———. Forthcoming. “Choreomusicology and Dance Studies: From Beginning to End.” Under Peer Review. Kress, Gunther. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. London and New York: Routledge. Kress, Gunther, and Theo van Leeuwen. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. Oxford: Oxford University Press. Locanto, Massimilian. 2018. “Choreomusicology Beyond ‘Formalism’: A Gestural Analysis of Variations for Orchestra (Stravinsky-­Balanchine 1982).” In Music-­Dance: Sound and Motion in Contemporary Discourse, edited by Patrizia Veroli and Gianfranco Vinay, 35–56. London and New York: Routledge. Minors, Helen Julia. 2006. “Paul Dukas’ La Péri as Interpreted by Two Balletic Collaborators.” The Opera Quarterly 22 (1): 117–35. ———. 2009. “La Péri, Poème Dansé (1911–12): A Problematic Creative – Collaborative Journey.” Dance Research: Ballets Russes Special Volume Part 2 27 (2): 227–52. ———. 2012a. “Music and Movement in Dialogue: Exploring Gesture in Soundpainting.” Les Cahiers de la Societe quebecoise de recherche en musique 13 (1–2): 87–96. ———. 2012b. “In Collaboration:Toward a Gestural Analysis of Music and Dance.” In Bewegungen zwischen Hören und Sehen: Denkbewegungen über Bewegungskünste, edited by Stephanie Schroedter, 163–80. Würzburg: Königshausen & Neumann. ———, ed. 2013a. Music,Text and Translation. London: Bloomsbury. ———. 2013b. “Soundpainting: The Use of Space in Creating Dance-­Music Pieces.” In Sound, Music and the Moving-­Thinking Body, edited by Marilyn Wyres and Lorenzo Glieca, 27–34. London: Cambridge Scholars Press. ———. 2016. “Mediating Cultures and Musics: An Intercultural Production of a Midsummer Night’s Dream.” In Routledge Handbook of Intercultural Arts Theory, Research and Practice, edited by Pamela Burnard, Elizabeth McKinley, and Kimberley Powell, 417–30. London and New York: Routledge. ———. 2019 (in press). “Opera and Intercultural Musicology as a Mode of Translation.” In Opera and Translation: Eastern and Western Perspectives, edited by Adriana Şerban and Kelly Chan. Amsterdam: John Benjamins. Morris, Mark. n.d. Dido’s Lament. Accessed December 10, 2018. www.youtube.com/ watch?v=ZrzC_KLI8KM. ———. 1995. Falling Down Stairs – Sarabande. Accessed December 10, 2018. www.youtube. com/watch?v=kAX7WF6iMcU. National Dance Company Wales. 2017. Programme ‘P.A.R.A.D.E.’ October 24–25. Wales Millenium Centre. Nijinsky, Vaslav. 1987. The Rite of Spring, reconstruction Millicent Hodson, Joffrey Ballet. Accessed May 20, 2018. www.youtube.com/watch?v=jF1OQkHybEQ. ———. 2013. Prelude à l’aprés-­midi d’un faune – reconstruction, Paris Opera Ballet. Accessed June 24, 2018. www.youtube.com/watch?v=4qjvGIMeIhU.

178  Helen Julia Minors

Satie, Erik. 2007. Parade, Picasso et la Danse, Festival Europa Danse. Accessed December 12, 2018. www.youtube.com/watch?v=YejpJ4kMH_0. ———. 2017. P.A.R.A.D.E., Carolyn Finn (choreographer), BBC 4, aired December 2. Accessed December 12, 2017. www.bbc.co.uk/programmes/b09f469j. Stravinsky, Igor. 1947. Le sacre du printemps. London: Boosey and Hawkes. ———. 2008. Le sacre du printemps, Vaslav Nijinsky (choreography), Millicent Hodson (reconstruction), Set/Costume (Nicholas Roerich), Marinsky Orchestra and Ballet, Valery Gergiev (cond.) [DVD] BelAir BAC241. Sturge, Kate. 2011. “Cultural Translation.” In Routledge Encyclopedia of Translation Studies, edited by Mona Baker and Gabriela Saldanha, 67–70. London and New York: Routledge. Susam-Saraeva, Şebnem, ed. 2008. “Special Issue: Translation and Music.” The Translator 14 (2). Venuti, Lawrence. 1998. The Scandals of Translation. London and New York: Routledge. Wenk, Arthur. 1976. Claude Debussy and the Poets. Berkeley: University of California Press. ———. 2012a. “Music and Movement: A View from Cognitive Musicology.” In Bewegungen zwischen Hören und Sehen: Denkbewegungen über Bewegungskünste, edited by Stephanie Schroedter, 151–62. Würzburg: Königshausen & Neumann. ———. 2012b. “Music, Dance, and Meaning in the Early Nineteenth Century.” Journal of Musicological Research 31 (2–3): 147–65. ———. 2018. “Ways of Knowing: Social Dance, Music, and Grounded Cognition.” In Music-­ Dance: Sound and Motion in Contemporary Discourse, edited by Patrizia Veroli and Gianfranco Vinay, 55–75. London and New York: Routledge.

8 WRITING DRAWINGLY A case study of multimodal translation between drawing and writing Tamarin Norwood

Introduction This chapter offers an anecdotal account of a particular method of “half-­blind” drawing (part i), from which a novel method of writing is derived (in part ii). This process of derivation can be understood as a form of multimodal translation in which a drawing method (the original “text” being translated) is extracted from its original mode of drawing and reconstituted in the mode of writing, producing a method of writing “drawingly.” The writing that results from this method is, for academic writing at least, formally and structurally unusual. Vertical ticks appear in the body of the text where digressions present themselves during the act of writing, and the possibilities of these digressions are drawn out and explored further along the text, creating linked lines of textual fragments that feel their way down and across the page. As they feel their way, these lines closely probe the object of the writing so as to gradually reveal its contours, in much the same way that the tip of the pencil probes the page and the eye probes the object in drawing. As Jean-­Luc Nancy observes, the object of a drawing is sought and gradually found through the gradual formation of its image on the page (Nancy 2013, 10). Through this process, he writes, one arrives at an idea of the object that one had not formed and could not have formed before the drawing began (10). This enquiry and gradual encounter is played out on the surface of the page, the “formative force” of each exploratory venture leaving on the page a visible record: a material concretion of the process that formed it (Nancy 2013, 12). I propose that writing produced drawingly can likewise be read as the material residue of the search it has undertaken; a residue produced almost incidentally to the process of writing but kept nevertheless, rather like the gestural line kept on the page of Tristram Shandy after the flourishing movement of Corporal Trim’s

180  Tamarin Norwood

stick (Sterne 1978 [1762], 743).1 Here, process – in writing as in drawing – takes precedence over product. Can the same be said of the particular process of multimodal translation I have employed? Has the translation process taken precedence over the text it has produced? Reflecting on his approach to translating Derrida’s What is a ‘Relevant’ Translation?, Lawrence Venuti chose to adhere “as closely as possible to his French, trying to reproduce his syntax, lexicon, and typography by inventing comparable textual effects – even when they threaten to twist English into strange new forms,” while balancing this strangeness against the need to maintain “a level of intelligibility and readability” for the English-­language reader (Venuti 2001, 173). Committing to a process of writing drawingly demands a similar balance between writerly intelligibility and close adherence to drawing praxis – only the prospect of twisting the writing into strange new forms presents itself more as an opportunity than as a threat. Open to this opportunity, my approach to translating between drawing and writing shares the myopic and closely probing perspective of the pencil tip that Derrida associates with the movement of the blind man who, in order to move at all, “must advance, advance or commit [himself], … run through space as if running a risk” (Derrida 1993, 5). Once I had identified the character and features of the half-­blind drawing method, and established the methodological parameters they implied for writing, what remained was to advance – to run the task of translation as though running a risk – and thus, returning to Nancy, to arrive at something that could not have been formed before the drawing (or the translation) had begun. In this way the text in part (ii) is not only the residue of a myopic writing process but of a myopic translation process too: a process of translating “drawingly.” Through this process the form of the writing, adhering as closely as possible to its origin in drawing, is indeed twisted into strange new forms. The experience of writing the text, which I hope is mirrored by the reader’s experience of reading it, was an effort of concentration not unlike my pencil’s close pursuit of the life model’s contours as they were interrupted by the junctions of his fingers, knuckles, and forearm in the anecdotal description that follows. Every detail of these interruptions needed following along their length until they gave way to the next digression and the next, and only in this way did a picture of my object – in drawing as in writing – gradually take shape on the page. If the example of multimodal translation in the present chapter is a process of translating “drawingly,” can we look further into the praxis of drawing for new ways of analysing the translation process? As the writing in part (ii) myopically felt its way across the surface of the page, questions of multimodal translation were not in sight and were never explicitly addressed. But the reader interested in the subject of translation will no doubt encounter junctions of my text that draw their attention towards digressions of their own, and by following the path of such associations that “arise in the mind unbidden,” they might arrive at a picture of translation that, following Nancy, had not formed before their reading had begun.2 For instance, the body of the life model might be imagined as an original “text,” with the pencil sketch its translation into drawing. Following the path of this

Writing drawingly  181

association, one might consider whether, by presenting his body in multiple poses, the model is offering multiple readings of his body, and whether each view of his body is a new reading of the same text or a new text altogether. One might follow the path further still and ask whether the eye of the artist moves across the body of the model as the translator’s eye moves across the body of the text; and further still, to ask whether artist and translator similarly aspire to a perfect correspondence of eye and hand, rendering their intervention perfectly imperceptible and their translation, or their drawing, perfectly equivalent to the original. And given that perfect equivalence is unattainable in drawing as in translation, one might probe further still, to seek in examples of translation practice the crisis point John Berger so often encounters when his drawings near their completion, when he “begins to draw according to the demands, the needs, of the drawing” (Berger 2005, 8). If at this point the drawing is “in some way true,” he goes on, “then these demands will probably correspond to what one might still discover by actual searching. If the drawing is basically false, they will accentuate its wrongness” (8). Can the same be said of a translation?3 One could go on, creeping further and further along this and other wandering paths that present themselves in the analogy between drawing and translation.4 Finally, what emerges from the writing process is a depiction of this point of contact as a myopic, wayfaring, and pathbreaking movement provoked above all by a desire on the part of the person drawing to draw up, use up, or consummate their object in the laying down of a graphite line. It is a movement depicted as slow, sensitive, yielding, and almost caressing, but also duplicitous in the way it approaches its object, seeming to caress it with the closeness of its touch only to capture it and, in capturing and pinning it onto the page, to obliterate whatever had been so uniquely ungraspable about it.5 If drawing is indeed an embodied impulse to caress and capture its object, taking in the risk of obliterating that object’s uniqueness in the process, can the same be said of translation, or at least of the example of translation offered in the present chapter? In translating it into a method of writing, have I approached my original text – the method of drawing – with the softness of a yielding caress that conceals the threat of obliteration and, if so, what is left of the original text? To these questions I would respond by returning to Derrida as he dramatises a personal account of drawing: “it is as if, just as I was about to draw, I no longer saw the thing. For immediately it flees, drops out of sight, and almost nothing of it remains. . . . It blinds me while making me attend the pitiful spectacle” (Derrida 1993, 37). The object’s flight at the very moment of its capture is, in the pitiful spectacle of drawing and perhaps also of translation, what guarantees the life of the original even after its contours have been captured (always imperfectly, if caressingly) onto the page. Left unconsummated at the conclusion of the drawing process is the “ineluctable presence – the thingness of the thing” (Schwenger 2001, 102) that remains untranslatable but is described by the search left behind on the page by the myopic and committed advance of the tip of the pencil, the tip of the eye, the tip of the translator’s task.

182  Tamarin Norwood

i Rolled up on my desk lie several scuffed sheets of cartridge paper holding the drawings and notes I made over a couple of autumn months four years ago. I had joined an evening life drawing class because I never seemed to draw anymore. I have not looked at the drawings much since, but I remember the class well. Each session opened with a couple of very quick poses, then some five-­or ten-­ minute exercises focusing on light, tone, or line, and then work would begin on a final half-­hour pose. Before embarking on this final drawing, we were reminded of how to mark up our pages to avoid making errors of measurement or composition that might only surface when it was too late to correct them.We should hold out our pencils against our view of the model, horizontally and then vertically, and use these impromptu rulers to measure with our thumbs a set of relative heights and distances between landmarks about his body: shoulder to knee, for instance, shoulder to toe, and head to navel to hand. The task was to plot this network of points onto the page and elaborate them into a composed and proportioned image that should resemble, in the end, what each of us saw of the model from where we were standing. The drawings I produced in this way always seemed to lack something, and it took me a while to understand what. Certainly I was out of practice, and to look at, the drawings were not all they could have been. But it was not a problem of looking; it was a problem of touch. The drawings lacked what I realised I must have come to the class to enjoy: the soft fur and scratch of the lead with the grain, the time spent against the page, the tugging and coaxing and scraping of form – there was none of this. Instead, before the encounter had even begun I had constructed an invisible apparatus of sight-­lines to articulate the gap between the model and my eye, and between my eye and hand and page: an apparatus that hardened the network of landmarked points and the features between them, extracted them, shifted them through the air and delivered them onto the paper. The apparatus seemed to get in the way, or seemed to thicken the air and make that get in the way, with the effect that the model’s body and my drawing were held apart by the very lines of sight that were meant to bridge them. The pencil darted about in the air like a scalpel or a beak, pecking at the paper rather than burrowing into it, and without those sensitive and spontaneous excavations, drawing with a pencil didn’t seem such a compelling thing to do. When I went back the following week, I drew in a different way. I looked only at the model, and never once at the page, until the drawing was done. I continued to work in this way, half-­blind as I thought of it, for the rest of the drawing course. It was engrossing to draw like this. It seemed to seal up the gap between the object and the page and the eye, contracting all the imaginary apparatus into one point of contact that sees, measures, and marks in a singular probing act which, best of all, is always only the burrowing activity that engages me most about pencil work. The way I worked was to plant my eye at some point on the skin of the model and at the same time plant the tip of my pencil at some point on the empty page, which from then on would refer to the corresponding point on the model’s skin.

Writing drawingly  183

Now paired against their surfaces, the eye would begin to root its way across the skin as the pencil began to root its way across the page. The tip of the pencil became the tip of the eye – a groping proboscis feeling its way about the soft form of the object and seeing only the little dot of its surface presently under description by the very end of the lead. As the eye moved, so did the pencil, as much as I could manage, and in this way the eye navigated the page by means of the blind point of the pencil, and drawing proceeded without my ever looking at the page nor ever losing contact with its surface. Some weeks I would try to make the drawing resemble the model, and to do this I needed to be more strategic. Once paired against the page and the model’s skin, pencil and eye had to move exactly in time with one another, because if either temporarily slowed down or sped up, inconsistencies of scale tended to be introduced. If they both moved too quickly, details might be missed that could not be revisited for correction later on, because I would be unable to find the place again. Every detail needed attending to at the very place and time it was first encountered. Burrowing about the surface of the page, the tip of the pencil maintained contact with the model’s body only by the contact it maintained with the paper, so if it came detached from the page, the body and the drawing would drop out of sight and could not be retrieved. Sometimes I was able to adjust marks laid down very recently and very close by. It turned out to be possible to retrace a route just taken provided the muscles of my hand could remember the last few flexes of the fingers or the last adjustment of the wrist, and were able to repeat this sequence of movements in reverse. The memory of the hand offered a couple of inches of revision – a second or so – and this redress could be put to use strategically. For instance, I might find the junction between the model’s knuckle and fingers comprises five paths short enough that each finger can be traced onto the page with each return trip to the knuckle brief enough and swiftly enough that the muscle memory of my drawing hand can render it all quite well. But it would be a different matter at more complex or multiple junctions of his body. What would happen when the forearm intersected the collarbone, escalated into a hand of its own and then needed returning to the collarbone to intersect it a little further along, such that the positions and angles of the arm and the remaining length of collarbone looked uninterrupted by my foray through the hand and back again? The muscles of my drawing hand would fail to remember a procedure as complex as this, and tides of error would be introduced. The form would end up flayed across the page, elements pivoting through one another at every junction. I would try alternative strategies: crawling the pencil along in a series of branching advances and retreats, or choosing the routes that might be salvaged by muscle memory and attending to them together, then accepting as inevitable that regions with sparse detail will disorient the pairing of pencil and eye, creating regions and contours that are irregular or indistinct; and that any route I choose might leave whole islands altogether unreachable and corresponding gaps on the page. I began to bring thread and masking tape with me to the classes, and later on blu-­ tac and drawing pins, too. With these I would construct lines, regions, and points to

184  Tamarin Norwood

guide my hand as it moved blindly about the page. It meant I would spend the first few minutes of each pose preparing the paper, estimating the centre point and marking it with a drawing pin, or taping thread onto the drawing board to bisect the page horizontally and vertically and give me four quadrants into which I could divide my composition. Sometimes I would measure the model with my thumb held out against my pencil, as we had in the first class, and plot certain landmarks on the page in blu-­tac before I began to draw, so I could elaborate the reference points later on by touch. The more intricate the apparatus, the more accurate a picture I was able to draw, and with practice I learned to obtain a drawing that quite closely resembled its object. The tip of my eye and the tip of my pencil were still paired as before, and they would still move about their respective surfaces in time with one another, but now my hand was obliged to the guides as well and had to get used to subtly stretching time or stretching space by speeding or slowing its movement. The correspondence of hand and eye had lost its closeness. It had become only one of a complex of correspondences that included the skin of the model and its relation to the flatness of the page, my memory of how I had divided up the page and its relation to my view of the model, and how all of these related to the feel of thread or other apparatus that would sometimes, unexpectedly or otherwise, cross my path. By this stage I had noticed I was losing interest in assembling these ever-­more complex guides. There came a point, I felt, that I might as well just be looking at the page. * Throughout the life drawing course I was engrossed, I believe, in the experience and sensation of drawing. But at the same time I was gripped by the detail of the drawing process. Every little aspect of the process seemed to me a completely new invention, and moreover seemed to bristle at every turn with obscure significance, as though every aspect were a metaphor for something unknown, or at least not visible from where I stood. Perhaps if I could stand somewhere else I would be able to see. Perhaps this is why I was so anxious to store in my memory every peculiarity of the process: the strategies, restrictions, and obligations, the precision of the matching of pencil and eye and surface and skin, and the ranging of touch and sight – what was missing and what was there in excess. Perhaps I wanted to store them so that I could be among them again at another time, from another vantage point. Whatever the reason, these reflections nagged and nagged at me until the effort to remember them, and the anxiety that I might forget them, began to distract me from the drawing altogether. When this happened, maybe two or three times in the course of a half-­hour drawing, I would stop work, scribble a note at the right-­hand edge of the page, and then resume drawing. Braver lines, reads one note: through space, | launching forth, | not blind | in the dark. Coming across such notes now, a few years after they were written, it is difficult for me to recall what I must have meant. In some of the notes, this difficulty is compounded by the handwriting, which is scrawling and ambiguous: was it launching forth, or launching form? Not blind, or not island? In any case I remember how these scrawls were put on the page. When I needed to write something down, I would fix my eyes very keenly on the model’s skin and, without moving the

Writing drawingly  185

pencil from its position on the page, I would feel my way towards its tip with my left hand, which until then would generally have been propped at the top edge of the drawing board to keep it steady. The left thumbnail would press against the tip of the pencil, oriented in the direction of its movement, and then, eyes still fixed in place, the pencil could be lifted up from the thumbnail marker and away from the page. From here it was easy enough to feel with my right hand the right edge of the page, scribble the words, locate the thumbnail again, and go on with the drawing in the direction indicated by its orientation. It took a greater effort of concentration to hold my eyes exactly still throughout the operation, and perhaps they did wander a little without my noticing. Little gaps or interruptions in the line were no doubt also introduced when I made these notes, but these were a small price to pay if I could shed the distraction of trying to keep everything in mind. But it would accumulate again, and more than once I stopped drawing altogether and left the studio early so no more would come. On the walk home, drawings rolled up under my arm, I would try to go through everything that had happened, and once home, with the drawings laid out across the table, I would consider everything at a pace I found acceptable: slowly and uninterrupted by the generation of even more. * Of course more comes anyway, even now that no drawing is going on. It seems to me that all this work has yielded two kinds of forms: the visible forms marked in pencil on the page and the invisible forms of the processes that had created those pencil marks. Both kinds of form tend to germinate and become increasingly proliferate the more time I spend with them. I can look at the marks on the page and identify the body parts I have drawn, even if some are ambiguous or sparsely described. But if I study patches of pencil marks close up, or squint and take in whole compositions at once, I can also see resemblances of things I had never knowingly drawn, like seeing particular pictures in clouds or blots of ink, but more concerted somehow. If I look at it long enough, the scribbled joint of a knuckle, for instance, resembles a copse of trees, then a root system or a junction of roads, then a clump of eyes or mouths or parts of ears, and now foliage again. One image shifts into another, sometimes swiftly and sometimes gradually and partially, so that distinct pictures might coexist or temporarily borrow one another’s parts. But the same is true of the other kind of forms – if form is really the right word to use – the invisible impressions and ideas that have collected around the drawing process. Although I cannot see their forms, they are palpable enough to grip hold of in my imagination, and I can spend time with them. As I squint and scrutinise them, I have found they bristle with resemblances in just the same way as the configurations of pencil they left on the page. The moving tip of the pencil becomes the tap of the seeing eye of a white cane, for instance, which taps a line or taps a reservoir that leaks and spills, or it grazes the surface of the ground at the risk of harming its ocular membrane, which becomes the scratched tip of the pencil again, and so on. These invisible forms and their analogues move through and across one another in my mind, coexisting, borrowing one another’s parts, reinforcing, changing, and tempering one another as I watch. As they continue to proliferate, the

186  Tamarin Norwood

bonds between them multiply and thicken and spread out, attaching to one another at the edges and from inside. One idea splits into many more, each of these diverting through paths of their own, some closer, some further away, some vanishing only to reappear, strange and familiar, later on. It seems to me, as I write, that these resemblances are to be taken seriously. If I can follow their proliferation I might hope to approach whatever kernel they circle around. There is no directing the course of their growth, but there is something familiar about the way they grow that leads me to the looming impression – a kind of heavy inkling – that they are all describing the nature of the very tip of the pencil and the particular kind of contact it makes as it moves about the page. Just this tiny thing. A dot. I am surprised it has enough mass to pull me towards itself.

ii A theologian loses his sight |. He learns to make his way | to work |. By sound, smell, the movement of air on his skin | and the tap of the cane against the pavement he learns to track his position on the ground with some precision. Meanwhile, his mind is in the air, so to speak, picturing himself as a moving point on a memorised | map of the route. As long as he can keep his movement along the ground and on the map in time with one another, he knows where he is. It becomes difficult when the two topologies suddenly cease to correspond. An obstacle on the pavement can throw him off course completely. With the white cane, he can move around small obstacles without losing his bearings, but with larger or more complex obstructions it is a different matter. A car parked across the pavement, or worse, one car after another parked at unexpected angles, are structures that risk introducing tides of error into his course as he tries to follow their perimeters back to the route he is taking. Turned through all these angles, he might still know more or less where he is on the memorised map, but not which way he is facing | (Hull 1990, 90–91). Loses his sight. His story reads like a parable, although the lesson it teaches is unclear. He learns to make his way. I imagine he makes his way more than I, who can see, make mine. To work. The object(ive) of his drawing. His walk produces not the line of his movement but his arrival at work. I wonder if his account sounds the way it does – conversational, open to possibility, feeling their way – because it is composed in the moment of speaking. Then again, perhaps I hear these qualities in his account only because I know he spoke it into a tape recorder. I would like to know if he spoke to an imaginary listener as he recorded, or just to the tape.

Writing drawingly  187

Memorised. I watched a film once of a child’s sight being restored. While he was blind he would race around the house, every orienting marker – doorknobs, lampstands, the plant pot on the windowsill – landing at his fingertips just as he expected. These were his pivots, and he would ricochet from them and navigate his home with reckless precision. The film showed him running down the stairs and out of the door, excited to get to the hospital for the procedure.   Once home from the hospital, his restored sight seemed to blank out what he knew about his surroundings. He stumbled and tripped, his turns were miscalculated, his feet and fingertips weren’t where he expected them. He had to learn to make his way around all over again. Blind, bind Many years ago I heard a woman on the radio telling the story of her grandmother leaping and dancing arabesques through the fields as a little girl, delighted that the proud day had come for her feet to be bound. The irony is so concise I now find myself wondering if I had misremembered, and it was fiction after all. Was it Emily Prager’s short story “A Visit from the Footbinder” (Prager 1999)? It describes much the same thing happening, only it was set in the China of 1260 and the child scampers around a palace, not the countryside, while her elder female relatives shuffle and walk with canes. Her story was criticised for perpetuating “the opposition between Western observer and Eastern ‘object,’ reinforcing a readerly politics of domination” (Newman 2007, 6). Exotics, erotics, “visual erotics allows the object of vision to remain inscrutable” (Marks 2000, 184). “The seeking of the caress constitutes its essence by the fact that the caress does not know what it seeks. This ‘not knowing,’ this fundamental disorder, is the essential. It is like a game with something slipping away, a game absolutely without project or plan, not with what can become ours or us, but with something other, always other, always inaccessible, and always still to come” (Levinas 1989 [1947], 51). Which way he is facing. There is a game children play. One puts on a blindfold and is spun around and around by the other children, slowly to begin with and then more quickly until she no longer knows which way she is facing. As she begins to stumble, the other children flee and she runs after them |, hands outstretched, following the sound of their voices until she has caught each one – or else gives up, unties the blindfold, and restores herself to sight. She knows that the knot is within her reach and she can untie it at any time |, but she also knows that if she does it will bring the game to an end.

188  Tamarin Norwood

The other children flee and she runs after them. Imagine all the children leaving traces of their movements on the ground, or a video camera set above them and the aerial view of their movements faithfully transcribed onto a page. The task of translating movement into mark would demand not only copying the lines of each child’s path but drawing something about their speed, the certainty with which they place their feet, the work of their hands and eyes. It would be easy to identify in such a diagram which child wore the blindfold and which were free. She can untie it at any time. “Naturally his eyes would be able to see. But they are blindfolded . . . not naturally but by the hand of the other, or by his own hand, obeying a law that is not natural or physical since the knot behind the head remains within a hand’s reach of the subject who could undo it. . ., as if he chose it, at the risk of a fall |” (Derrida 1993, 13, emphasis original). Fall. I have wondered whether he means “fall” in the biblical sense too, when things were first lost to words. One of my half-­blind drawings troubles me. It’s not the fact that the leg looks mauled and out of joint that offends me (after all, the jaw looks fractured and pulled away from the skull and I don’t mind this); it’s that the leg looks wrong in the wrong way. The rest of the body is compelling, precise, sensitively explored, and the many anomalies resulting from my blindness on the page coax from the body a heightened sensitivity of its own. She seems to be lost in thought. Then a leaden foot is planted, her only contact with the ground, the pencil ragged now and nearly blunt. It gets the spacing wrong, and the spacing looks all the more wrong because the angles are fairly right. I know I drew the legs last. Perhaps I had lost interest by then. Perhaps I knew by touch or sound that the lead had got too thick and its lines would be bad for the body, but I wanted the body complete, so I went through the rest without much care. Then again, perhaps the trouble is not the drawing but the fact that I’m looking at it. Perhaps the trouble is that I’m looking at it and wanting it to look good. The tip of the pencil is something like an eye moving along the surface of the page |.With a focal length of nil, it watches the black deposit of graphite dragging | from its tip and adhering to the page, seeming all the blacker because there is no light to see by at such close range. There is always only this | black. Whatever it draws | onto the page, however the speed and orientation of the line might change, the little spot under the scrutiny of the pencil’s eye varies very little at all. The surface of the eye collects scratches | and marks as it grazes | the page, of which some must surely damage its sight | – sight so blunt it is nearly touch |, sensing nothing of the ground behind it or ahead |. To see what lies ahead it must commit to drawing forward into the dark, to running through space as if running a risk |.

Writing drawingly  189

An eye moving along the surface of the page. Derrida imagined what the pencil might see, pressed so close to the page, and he found it deeply myopic. From the “aperspective of the graphic act,” the stylus is blind both to the form that precedes it (the thing it sets out to draw) and the form that follows it (the drawing that will appear on the page) with the result that it sees nothing but the immediate present of its “originary, pathbreaking moment” (Derrida 1993, 45). The black deposit of graphite dragging. “Here |, to trace is to find, and in order to find, to seek a form to come (or let it seek and find itself) – a form to come that should or that can come through drawing” (Nancy 2013, 10). Here. Here, on this page, too. This. This, the black of this ink, too. Whatever it draws. Or writes. Imagine that when you write there is always only this black: the myopic tip of the pen, or the short-­sighted eye of the I-­beam cursor, blinking (I . . . I . . . I. . .). The surface of the eye collects scratches. I wonder if it hurts. And if it hurts the page as it grazes the surface. As it grazes. “Consider the cows, grazing as you pass by; they do not know what is meant by yesterday or today, they leap about, they eat, rest, digest, leap about again, and so on from morning until night and from day to day, fettered to the moment and its pleasure or displeasure, and thus neither melancholy nor bored. This is a hard sight for man to see; for he thinks himself better than the animals because he is human, but he cannot help envying them their happiness – what they have, a life neither bored nor painful, is precisely what he wants, yet he cannot have it because he refuses to be like an animal” (Nietzsche 1997 [1876], 60). Some must surely damage its sight. “Mit allen Augen sieht die Kreatur das Offene. Nur unsre Augen sind wie umgekehrt und ganz um sie gestellt als Fallen, rings um ihren freien Ausgang. Was draußen ist, wir wissens aus des Tiers Antlitz allein; denn schon das frühe Kind wenden wir um und zwingens, daß es rückwärts Gestaltung sehe, nicht das Offne, das im Tiergesicht so tief ist.” (With wholly open eyes a creature sees/the Open. Our eyes alone are as if turned about and wholly ranged about/as traps that shut its clear way out off.// We sense what lies outside from animal/countenance alone; newborns even/we turn about and force them on their backs/see artifice and not the Open, set/ deep within the animal.) Rilke 1997 [1923], translation by Anton Viesel

190  Tamarin Norwood

Sight so blunt it is nearly touch. As he constructs an analogy between drawing and blindness, Derrida writes less about the absence of sight than the presence of touch. The stylus is a staff of the blind that feels its way, providing a tactile proxy for sight, and the fingertips grope about the page “as if a lidless eye had opened at the tip of the fingers, as if one eye too many had grown right next to the nail, a single eye, the eye of a Cyclops or one-­eyed man”; this lidless eye is “a miner’s lamp” whose light makes inscription possible. Throughout his account of blindness, the conflation of movement, touch, and sight is never out of reach (Derrida 1993, 3). Sensing nothing of the ground behind it or ahead. “Proceeding on our way things fall into and out of sight, as new vistas open up and others are closed off. . . . Thus the knowledge we have of our surroundings is forged in the very course of our moving along them” (Ingold 2007, 87–88). Running through space as if running a risk. Derrida describes the blind men in the drawings of Antoine Coypel. “Like all blind men, they must advance, advance or commit themselves, that is, expose themselves, run through space as if running a risk. They are apprehensive about space, they apprehend it with their groping, wandering hands; they draw in this space in a way that is both cautious and bold; they calculate, they count on the invisible. It would seem that most of these blind men do not lose themselves in absolute wandering” (Derrida 1993, 5). Of the many allegorical drawings of blind men | in Memoirs of the Blind, some move forward alone, with no cane and with arms outstretched, fingers groping ahead | of themselves. They are deeply closed into the dark, into the touch of the immediacy of the place: a picture of exquisite closeness | and excess of sensation |. This must be a kind of wandering, and it is portrayed as nothing good |. The blind men in these pictures have in their minds’ eyes not a map of their surroundings but a hope of sight restored. This is why these men are drawn: to show the miracle of vision, coming into the light, the salvation of enlightenment |. Allegorical drawings of blind men. There are many such drawings printed in Memoirs of the Blind (Derrida 1993, 5). Fingers groping ahead. Ahead. “To anticipate is to take the initiative, to be out in front, to take (capere) in advance (ante). Different than precipitation, which exposes the head | (prae-­caput), the head first and ahead of the rest, anticipation would have to do with the hand. The theme of the drawings of the blind is, before all else, the hand. For the hand ventures forth, it precipitates, rushes ahead, certainly, but this time in place of the head, as if to precede, prepare, and protect it. . .  Anticipation guards against precipitation” (Derrida 1993, 4). Exposes the head. Sometimes, wearing a scarf on cold days, it occurs to the blind theologian that he could equally go about with his whole head covered, not just the lower part of his face (Hull 1990, 53).

Writing drawingly  191

“And so with my forehead I ran against the wall thousands and thousands of times, all day and all night long, and I was glad when I banged myself bloody, for this was proof that the wall was beginning to harden” (Kafka 2007 [1931], 165). Vertiginous somersault, perilous freefall, bang your head bloody, graze your eyes. A picture of exquisite closeness. For example, Antoine Coypel, Study of the Blind, Louvre Museum. How wide open is his eye, how much surface it has. And excess of sensation. “I suspect, nevertheless, that he was not very capable of thought. To think is to forget a difference, to generalize, to abstract. In the overly replete world of Funes there were nothing but details, almost contiguous details” (Borges 1985, 104). Portrayed as nothing good. The blind lead the blind into a pit: underground. Luke 6:39. The salvation of enlightenment. Laura U. Marks writes about touch and the occlusion of sight and points out that lacking sight does not mean lacking insight. She writes that the analogy between seeing and knowing depends upon a conception of sight roundly questioned by the feminist and phenomenological lines of thought that support her account (Marks 2000, 133). Because it can perceive over distances, she writes, sight has come to be understood as the sense most separate from the body, the most cerebral and the most suited to objective observation, in contrast to non-­visual senses consequently thought of as natural, prediscursive, or not cultivated (132). And from there, the disembodied, all-­seeing eye of vision has become associated with the objectifying gaze that propagates a separation between subject and object (162). There are other ways to attend to this separation, Marks writes, if we can approach it by means of insight brought about by the occlusion of sight (191–93). No roots are put down when moss grows |, so patches can be lifted up and gathered with the hands or by sliding a trowel just underneath the surface of the soil. It is sometimes possible to pull it up in handfuls, like pulling hair, though this is a delicate manoeuvre which must be done slowly so that you can monitor the many resistances of the rhizomes it nests in the soil and vary the direction of your movement so you tear as few as possible.You can hear the low sound of fibres breaking in the soil as you tug. Once collected, the fragments can be held in the palm of the hand | and carried to the garden |, where they can be arranged on ground prepared in advance |. Once positioned, the surface must be watered deeply and pressed down by treading over it with your feet, and if necessary, patches should be held in place with vertical metal pins |. No roots are put down when moss grows. Moss is a rhizome.“Unlike trees or their roots, the rhizome connects any point to any other point. . . . It has neither beginning nor end, but always a middle (milieu) from which it grows and

192  Tamarin Norwood

which it overspills. [Its] lines, or ligaments, should not be confused with lineages of the arborescent type, which are merely localizable linkages between points and positions” (Deleuze and Guattari 2004, 21). The fragments can be held in the palm of the hand. Held in the hand, a tuft of acrocarpous moss resembles a small rodent |, and can be stroked from nose to tail. Resembles a small rodent. “A fragment, like a small work of art, has to be entirely isolated from the surrounding world and be complete in itself like a hedgehog” (Schlegel 2002 [1798–1800], fragment 206). Appraising the way the fragment was used by the Jena romantics, Philippe Lacoue-­Labarthe and Jean-­Luc Nancy expand upon “what should be called the logic of the hedgehog” (1988, 44). Fragment 206, they write, compares the fragmentary work to a “small work of art,” having an individuality and totality of its own, but being “neither directly nor absolutely the Work” (43). Because it is a genre of plurality – that is, “to write the fragment is to write fragments” – an individual fragment is both complete in itself and indicates the completeness of the whole of which it is part (43–44). Carried to the garden. You could leave moss where you find it on the forest floor and still cultivate it. I wonder, then, whether it would still be the forest floor or a garden. Ground prepared in advance. To cultivate a moss garden from scratch, the area must first be cleared of all vegetation and plant debris, the land smoothed to encourage branching and rhizome attachment, and contoured if necessary to eliminate any small depressions that might collect water. The surface soil must be loosened slightly with a rake just before the moss is laid down, to help it make contact with the ground. The area cleared and smoothed, and lightly raked: I think of the presuppositionless starting point Edmund Husserl described when he began to work on phenomenology (1970, 263). Imagine lightly raking the surfaces of certain phenomena and lightly raking the surface of the mind that might brush against it. Imagine these surfaces bristling warmly in a manner that is almost feline, the way a cat presses the arch of its back against your hand to redouble your caress as you stroke it. Held in place with vertical metal pins. The “logical and organized garden,” writes Jyrki Siukonen, is an enclosed place, familiar and cultivated – the place before the Fall – and it stands in contrast to the unorganised, hostile, even sublime forest with its limits unknown (1996, 2). “Attempts to understand the forest with the logic of gardening,” he writes, “are doomed to fail. Gardening is, especially in its domestic forms, something that can be

Writing drawingly  193

learned from books, while in the forest you need knowledge that can only be learned by living in the forest” (2). In the forest “lies the root of poetical knowledge,” something which, like intuitive knowledge, resists being clearly understood by logical thought (2). He describes one such effort as an attempt to “clear away all obscurity and create an organised and reliable system of knowledge . . . by cutting off all ‘overgrown’ branches and end[ing] up with a tree of knowledge pruned into exact mathematical shape, like those seen in formal gardens” (3). A film by Shauna Beharry, Seeing is Believing, shows a photograph of a piece of fabric, shot | at very close range (Beharry 1991). The closeness means the shot is often out of focus, and is very dark indeed when the lens makes contact with the photograph. When the image is dark, it becomes grainy and even harder to see. | Watching the video, most of all you are aware of surfaces: the surface of the lens, of the photograph and of the fabric, of the screen upon which the video plays, even the surface of your eyes. | Of watching the film, Laura U. Marks writes: “I realize that the tape has been using my vision as though it were a sense of touch; I have been brushing the (image of the) fabric with the skin of my eyes, rather than looking at it” (2000, 127). Each surface is pressed so near to the next that there is never enough distance to see the whole of any object, to understand its geography, to understand the route being made across the terrain. You see it so closely that it evades your vision, but neither is there anything to touch. | Shot. Words also murder things (Schwenger 2001). Grainy and even harder to see. Beharry is “using video to show the limits of vision” (Marks 2000, 112). “The caress of the eye over the skin is so utterly, so extraordinarily gentle, and the sensation is so bizarre it has something of a rooster’s horrible crowing” (Bataille 2001 [1963], 66). But neither is there anything to touch. Marks writes about haptic visuality in the context of diaspora film and video, where the haptic answers a longing to return to the sensual experience of inhabiting an environment you have left behind. Where you can no longer touch the skin of a loved one, she argues, you can use the lens to engage with the touch of skin while engaging with the absence and incommunicability of that touch. The touch of lens against fabric is what makes the fabric inaccessible to sight: it is “so engaged in the present that it cannot recede into cognition” (Marks 2000, 191). Meanwhile, as much as haptic images “might attempt to touch the skin of the object, all they can achieve is to become skinlike themselves” (192). In this way haptic visuality affords a way of knowing that “implies a fundamental mourning of the absent object or the absent body, where optical visuality attempts to resuscitate it and make it whole” (191).

194  Tamarin Norwood

The films and videos she describes create the conditions for haptic visuality in order to inspire “an acute awareness that the thing seen evades vision”: to acknowledge its unknowability, and keep its unknowability intact by establishing a relationship not of mastery but of mutuality (191). In this “bodily relationship between the viewer and the image,” “it is not proper to speak of the object of a haptic look as to speak of a dynamic subjectivity between looker and image” (164). Our neighbour had always been blind. He knew the roads around us very well. I believe the maps in his mind were drawn not from the air but from the ground: a series of memorised cues that told him the story of his route as he encountered them one by one. On the way home he would arrive at his gate, walk half a dozen steps further to our gate next door, tap it with his cane, and then turn back to his own gate and push it open. It was a habitual gesture, as practised and perfunctory as turning a key in a lock, but however briskly he seemed to execute the move, he would never omit it. Our gate must have been the final orienting cue. On a page of Tristram Shandy a fictional | Corporal waves his stick in a flourishing gesture of freedom and spontaneity (Sterne 1978 [1762], 743). The gesture itself is not quite free nor exactly spontaneous; it is an expression of these qualities, exerted upon the air to complete a spoken utterance about how freely a man can live outside the confinement of marriage. Fictional. Ingold’s description of the line indulges the fiction of the novel, imagining Corporal Trim’s gesture really to have preceded that of the author who simply copies it: “Like any other gesture, the Corporal’s flourish embodies a certain duration. The line to which it gives rise is, therefore, intrinsically dynamic and temporal. When, pen in hand, Sterne recreated the flourish on the page, his gesture left an enduring trace that we can still read” (Ingold 2007, 72). But Trim’s flourish is not like any other gesture. It is imaginary, and at some point in the process of its being imagined it is drawn over with the near-­identical gesture of the one who imagined it. I wonder which is more real, the imaginary line of the flourish or the visible one on the page? I wonder which came first: whether there really was an image of the Corporal’s gesture in Sterne’s mind first of all, which he faithfully transcribed onto the page; and if so, whether it was an image of the line he was to draw, or an image of the movement he was to make.When he saw the line on the page, was it new to him? Otherwise, perhaps the author slipped through the surface of the fiction for a moment, so to speak, and embodied his character, and the pen embodied his stick. He might have propped his manuscript upright on an easel, adjusted his grip on the pen, readied himself, and then struck: a performance.

Writing drawingly  195

In any case, once the line was on the page: then, afterwards, that was how the Corporal had moved his stick. But which way around are we to imagine it? Are we to imagine Trim standing on our side of the paper, facing away from us towards the page; or facing towards us, in which case he is behind the page, or within it, so that the page contains him? What about the inhabitants of the empire depicted in Umberto Eco’s imaginary map on a scale of one to one (Eco 1995)? If such a map should be spread out to cover the entirety of the terrain it depicts, are we to imagine the inhabitants of that terrain walking about underneath the map and consult it by looking up, or walking about upon the surface of the map, consulting it by looking down? In the latter case, as Eco points out, they would “in reality, inhabit the map” and not the terrain at all (98). Imagine the impossibility of producing a translation on a scale of one to one, its surface so similar to and so closely overlaying the terrain of the original text that the two are all but indistinguishable. A translation so perfect it need barely exist. What would be left of the original text? Would it degenerate, or at least be altered, by so close a translation, just as a terrain closely covered by the surface of its map would gradually deteriorate for lack of sunlight and precipitation (Eco 1995, 98)? And if so, just as the map would cease to faithfully represent the terrain as its ecology of adapts to new conditions, would the translation likewise begin to fail? I imagine tapping a gate as you might tap a tree: drilling a hole into the surface until the sap begins to bead and drip, and then inserting a spile to direct the fluid down the line. Or tapping the skull to alleviate pressure, or tapping a line to listen in. He taps the gate, extracts from it his route and takes it home.

Notes 1 Indeed, might all writing be read in this way, to varying degrees? 2 Peter Schwenger explores the generative potential of “pay[ing] attention to images that flicker so briefly at the borders of reading that we are scarcely aware of them” and his reminder that “associations arise in the mind unbidden, so that it is almost never possible to read at the denotative level alone” (2012, 25, 30). His account of reading, here informed by jazz improvisation as much as literary theory and cognitive science, has much in common with the tensioned myopia and digressive expansiveness of writing (and reading?) drawingly. 3 Can the same be said, indeed, of analogy between drawing and translation, of which certain paths of association might be “in some way true” and others “basically false”?

196  Tamarin Norwood

4 Developing the possibility that certain lines of enquiry might be more “true” or more “false” than others, Schwenger provides an insightful discussion of the “uses” and “pleasures” of any reader’s “superfluous associations” in the reading of any text (2012, 27). 5 This characterization of drawing bears resemblance to the Biblical story of the origin of language, in which Adam’s first act of naming transforms a thing into the object of a speaking subject. Naming the animals of Eden “nullified them as beings on their own account” (Hegel 1979, 221) and assimilates their being with the human conception of that being. Schwenger outlines the tensioned relationship in literature between the desire to capture and the risk that capture might “murder” its object (2001); alternative strategies for keeping the unknowability of the object intact by establishing a relationship not of mastery but of mutuality through the caressing look of “haptic visuality” in film and video are considered by Marks (2000).

References Bataille, Georges. 2001 [1963]. Story of the Eye. London: Penguin Classics. Beharry, Shauna. 1991. Seeing is Believing. Video. Montréal, Québec: Groupe Intervention Vidéo. Berger, John. 2005. Berger on Drawing. Cork: Occasional Press. Borges, Jorge Luis. 1985. “Funes the Memorius.” In Fictions. Translated by Anthony Kerrigan, 97–106. London: John Calder. Deleuze, Gilles, and Guattari, Felix. 2004. A Thousand Plateaus: Capitalism and Schizophrenia. Translated by Brian Massumi. London: Continuum. Derrida, Jacques. 1993. Memoirs of the Blind. Translated by Pascale-­Anne Brault and Michael Nass. Chicago: University of Chicago Press. Eco, Umberto. 1995. “On the Impossibility of Drawing a Map of the Empire on a Scale of 1 to 1.” In How to Travel with a Salmon and Other Essays. Translated by William Weaver, 95–106. London: Harcourt. Hegel, Georg Wilhelm Friedrich. 1979 [1802]. System of Ethical Life and First Philosophy of Spirit. Translated by H. S. Harris and Malcolm Knox. Albany, NY: State University of New York Press. Hull, John M. 1990. Touching the Rock: An Experience of Blindness. London: SPCK. Husserl, Edmund. 1970. Logical Investigations. Translated by J. N. Findlay. New York: Humanities Press. Ingold, Tim. 2007. Lines: A Brief History. New York: Routledge. Kafka, Franz. 2007 [1931] “The Burrow.” In Kafka’s Selected Stories, translated and edited by Stanley Corngold. New York: W. W. Norton and Company. Lacoue-­Labarthe, Philippe, and Jean-­Luc Nancy. 1988. The Literary Absolute. Translated by Philip Barnard and Cheryl Lester. Albany, NY: State University of New York Press. Levinas, Emmanuel. 1989 [1947]. “Time and the Other.” Translated by Richard A. Cohen. In The Levinas Reader, edited by Séan Hand, 37–58. Oxford: Blackwell. Marks, Laura U. 2000. The Skin of the Film. Durham, NC and London: Duke University Press. Nancy, Jean-­Luc. 2013. The Pleasure in Drawing. Translated by Philip Armstrong. New York: Fordham University Press. Newman, Judie. 2007. “The Readerly Politics of Western Domination: Emily Prager’s ‘A Visit from the Footbinder’.” Journal of the Short Story in English 48: 1–12. Nietzsche, Friedrich. 1997 [1876]. Untimely Meditations, edited by D. Breazeale, translated by R. J. Hollingdale. Cambridge: Cambridge University Press.

Writing drawingly  197

Prager, Emily. 1999. “A Visit from the Footbinder.” In A Visit from the Footbinder and Other Stories. London:Vintage. Rilke, Rainer Maria. 1997 [1923]. Duineser Elegien. Munich: Deutscher Taschenbuch Verlag. Schlegel, Friedrich. 2002 [1798–1800]. “Athenaeum Fragments.” In Classic and Romantic German Aesthetics, edited by J. M. Bernstein, 246–60. Cambridge: Cambridge University Press. Schwenger, Peter. 2001.“Words and the Murder of the Thing.” Critical Inquiry 28 (1): 99–113. ———. 2012. At the Borders of Sleep: On Liminal Literature. Minneapolis, MN: University of Minnesota Press. Siukonen, Jyrki. 1996. On Artistic Knowledge: Notes for a Minor Platonic Exercise. Leeds: The Centre for the Study of Sculpture. Sterne, Lawrence. 1978 [1762]. The Life and Opinions of Tristram Shandy, Gentleman. Edited by Melvyn and J. New. Gainesville, FL: University Press of Florida. Venuti, Lawrence. 2001. “ ‘Introduction’ to Jacques Derrida, ‘What Is a ‘Relevant’ Translation?’ ” Critical Inquiry 27 (2): 169–73.

BEYOND WORDS Concluding remarks Ángeles Carreres and María Noriega-­Sánchez

The chapters in this book have explored the interplay between translation and multimodality from a range of viewpoints and disciplinary-­theoretical paradigms. The scope and nature of the questions they have asked, and the centrality of those questions to both multimodality and translation studies, are in themselves testimony to the need for further cross-­disciplinary dialogue. It is not surprising that, in setting out to map the complex connections between multimodality and translation, the contributions to this volume have asked as many questions as they have answered. In so doing, they have helped establish a set of coordinates that can serve as a grounding framework for future research. Placed under close scrutiny, some of the key metaphors that have traditionally helped us conceptualise translation appear suspect when considered from a multimodal perspective. In particular, the notion of meaning transfer – built into the etymology of the word “translation” itself – that underpins much of Western translation theory, is perceived as inadequate in a social semiotics framework, as Kress has emphasised in his chapter. Writing within literary studies, Reynolds reaches a similar conclusion: notions of meaning transfer perpetuate the fiction of languages as wholly discreet entities that can be mapped onto one another in seamless equivalence. Metaphors of meaning re/trans-­positioning, re-­constitution, or re-­creation are felt to capture more accurately the processes at play in multimodal communication as well as in translation. Connected to this, the question presents itself as to what sorts of meaning reconstitution/transposition can legitimately be called translation without overstretching the term to the point where it would cease to be useful, and it is no surprise that several contributors to this volume have focused on this crucial issue (e.g., Kress, Kaindl, Tomalin, Minors). As may be expected given the inchoate state of research in the area of translation and multimodality, discussions on and around terminology have featured in a number of chapters (Kaindl, Kress, Tomalin). While the need for translation studies

Concluding remarks  199

to engage with multimodal communication is recognised throughout this volume, there is no consensus as yet regarding the terms used to name the various kinds of meaning reconstitution. We find two distinct takes on this question: one sees the need for specific terminology to denote instances of meaning reconstitution taking place across modes (e.g., transduction, transposition), as opposed to within the single mode of language (Kress); the other opts for retaining the term “translation” for all cases of meaning reconstitution, and to distinguish between intra-­and intermodal translation (Kaindl). While the use of a specific term has the advantage of focusing attention on the multimodal element, it also risks entrenching an outdated, language-­centred view of translation which – as several chapters in this book argue – it is time for us to move past. A more relevant notion of translation understands it as a practice of cultural mediation in which texts are transposed in terms of a range of variables: specifically, mode, medium, and genre, in Kaindl’s definition. Beyond the terms we choose to employ, the need to open up the field of translation – and of language – to an understanding of its multimodal dimension is one of the key tenets of this volume, and one that should inform future research in the field. Indeed, the centrality of language to human communication and, by extension, to translation is one of the long-­standing assumptions that contributors to this volume have sought to destabilise. As is richly illustrated in the various chapters, in today’s hyper-­connected world we find ourselves unprecedentedly immersed in processes of multimodal meaning-­making.Yet this should not obscure the fact that language itself is, and has always been, inherently multimodal (Reynolds), and that, therefore, even “language-­centred” translation practices need to engage with, and account for, the multimodal dimension. In relation to this, it is important to bear in mind that the notion of mode remains – for good reason – a functional, relative one. What one community regards as a distinct mode may be subsumed within a different mode in a different community. What functions as a mode in a certain kind of exchange for a certain purpose may not do so when either the context and/ or the communicational purpose shifts (Kress, Reynolds, Tomalin). This understanding of mode as a functional entity has served social semiotics well. However, translation comes to complicate the picture – in Kress’s words: Social semiotic theory assumes a homology of modes in the source society matched by a homology of modes – distinct and different – in the target society. The difficulty lies in the likely absence of such apt transpositional resources by modes in the target society. (p. 31, this volume) This difficulty of mapping the modes of one culture onto another is also highlighted by Adami and Ramos Pinto in their chapter. In outlining a joint research agenda for translation and multimodal studies, they point to the need to study specific modes and meaning-­making practices in order to understand their degree of codification and how they can be shown to correspond or not with linguistic and national boundaries. However, even as they advocate this agenda, they warn

200  Ángeles Carreres and María Noriega-­Sánchez

against the dangers of over-­codification. Mapping semiotic competence onto social groups is not only an extremely complex endeavour but also one that risks bringing about undesirable political outcomes (e.g., it may exacerbate certain social and cultural hierarchies, or result in the constriction of meaning-­making activity in a given social setting). Responding to this increasing realisation that multimodal analytical frameworks are required, new research methods have been developed in recent years to undertake the analysis of multimodal outputs, such as multimodal transcriptions and digital multimodal corpora (Pérez-­González 2014b, 127). While this preliminary work has already had a significant impact, further methodological advances are required. As Adami and Ramos Pinto argue in their chapter for this volume, most research methodologies (e.g., data-­collection methods) and means of academic dissemination are still heavily logocentric. Academic research needs to develop new ways of eliciting data and disseminating findings that go beyond verbal output and formats. In this sense, Norwood’s chapter in this book offers a bold experiment in methodological multimodal translation. Applying a method taken from the field of drawing to writing, Norwood’s work uses language to encourage us to move beyond the merely linguistic, and hence enacts precisely the kind of procedural innovation Adami and Ramos Pinto advocate. While elaborating a general research agenda for translation and multimodal studies, Adami and Ramos Pinto also highlight the need to strike a careful balance between large-­scale quantitative investigations and in-­depth fine-­grained qualitative research. Their suggestion of pursuing an integration with ethnographic approaches, which is already taking place in social semiotic multimodal analysis, could certainly yield fruitful methodological innovations for translation studies. In wider terms, research would be envisaged as a participatory endeavour to include “citizen socio-­semioticians” and “citizen translation scholars” (Adami and Ramos Pinto, p. 88 in this volume). As should be apparent from the discussions offered in several of the chapters in this book, collaboration is often crucial to multimodal creativity and multimodal translation. Minors, in particular, emphasises this point in her analysis of works involving music and dance. Collaboration, manifested in the convergence of industrial and amateur subtitling practices, also constitutes a cornerstone of the performance of citizenship, as discussed by Pérez-­González in his chapter. In the products resulting from fansubbing and other participatory media forms (e.g., the Chinese danmu), the translators-­commentators are anything but invisible; on the contrary, the desire to make their aesthetic, ethical, or political agendas visible, thus countering established narratives and discourses, clearly drives their interventions (Pérez-­González). These forms of multimodal experimentation open up new spaces – both aesthetic and political – in which a range of meaning-­making practices come together and interact in intricate ways. The translation of multimodal ensembles such as these calls for a revision of established translation practices and methodologies. The insights gained from a deeper understanding of this phenomenon will, in turn, have far-­ reaching consequences for the field of translator training. Translation didactics is still mostly focused on linguistic transfer competences (Kaindl 2013, 266), but that

Concluding remarks  201

model is no longer adequate. Although professional translators cannot be expected to double up as graphic designers, visual artists, dancers, software developers, and so on, a more comprehensive approach to collaborative translation competence that includes a multimodal awareness should be fostered. As the foregoing summary suggests, translation studies’ gradual move away from monomodal perspectives increases the need for more genuinely interdisciplinary interactions. The chapters in this volume have analysed a range of specific instances of multimodal translation, thus providing useful insights that can be applied to a wider range of multimodal outputs, such as children’s books, comics, plays, operas, video games, advertisements, animated poetry, spoken word performances, hypertextual fiction, dialogue interpreting, to name just a few.1 It is worth noting that audiovisual translation, and in particular film subtitling, has already benefited greatly from the application of multimodal theory (Taylor 2004, 2013, 2016; Gambier 2006; Pérez-­González 2014a), and forms of assistive mediation, like audio description for the visually impaired and subtitling for the hard of hearing, are thriving areas of research (Snyder 2005; Díaz Cintas, Matamala, and Neves 2010; Fryer 2016; Remael and Reviers 2018). The social semiotic emphasis of recent thinking about multimodality has been highly beneficial in focusing attention on the individuals or groups who determine which signs are deployed and combined in these kinds of communicative exchanges, especially those in which ensembles are converted into ensembles. As Kress puts it, “[s]ign-­makers and their agency as social actors are in the foreground and with them the social environments in which they make signs” (Kress 2010, 34). Nonetheless, future research will need to determine how social semiotic theories of multimodality (which are based on a notion of human social interaction) cope with the ever-­increasing phenomenon of non-­social (i.e., non-­human-­ produced) translations. Machine translation systems (such as Google Translate) convert a source text into a target text automatically, so translation – in one form or another – undoubtedly occurs. Yet the translation is not directly produced as a result of human social semiotic activity (though technologists were involved in writing the underlying software). And when a machine translation system is combined with an image-­generation system, then the resulting “system of systems” creates multimodal ensembles (i.e., combinations of written text and visual images). Once again, while these technologies are designed by humans who certainly have agency as social actors, the systems do not have agency in the same sense. They make decisions that have social consequences (e.g., they output sequences of words and images that convey meanings), but these are determined by mathematical models such as recurrent neural networks – and the latter generate the output. Systems of this kind clearly lack consciousness and an understanding of the meanings they generate, whether expressed via visual designs or linguistic structures. Nonetheless, since they are undoubtedly able to create meaningful multimodal ensembles, they can and do function as social actors (in some sense), and therefore they can have a significant impact on the social semiotic environments in which they operate. With the burgeoning use of virtual personal

202  Ángeles Carreres and María Noriega-­Sánchez

assistants like Siri, Cortana, and Alexa, such scenarios are becoming increasingly prevalent in contemporary society, and therefore they merit greater consideration in discussions of multimodal communicative exchanges when translation provides the primary analytical focus.2 Writing in 2014, Pérez-­ González predicted that multimodality “is bound to become even more central to translation scholarship in future years” (Pérez-­ González 2014b, 127), and echoed Tymoczko’s claim that, as part of this process, the field of translation studies would need to engage in the re-­theorisation of some of its principles (Tymoczko 2005, 1090). Key to that endeavour is the re-­ conceptualisation of translation as a process that involves not just – and often not even primarily – language, but also a broad range of modes and semiotic resources. While translation studies has made incipient, but significant, attempts in recent years to engage with the field of multimodal studies, this interest has not been extensively reciprocal. The re-­positioning of translation as a process of cultural mediation that engages with multimodality not as a curious add-­on, but as part of its normal operation, will go a long way towards making translation – and translation studies – more relevant to multimodality. By bringing together expertise from both fields, this book has sought to encourage the re-­theorisation of translation studies along multimodal lines, while also alerting proponents of multimodality to the potential for interdisciplinary dialogue that the theory and practice of translation offer.

Notes 1 A recently published special issue of Linguistica Antverpiensia (Vol. 17, 2018), entitled Methods for the Study of Multimodality in Translation, edited by Catalina Jiménez Hurtado, Tiina Tuominen, and Anne Ketola, illustrates this sweeping scope with articles that range from webcomics and GIFs in political satire to audio description for art museums, and multimodal patient information guides (Jiménez Hurtado et al. 2018). 2 Multimodal machine translation is an emerging and promising field, as demonstrated by a number of recent publications and conferences (Shah, Wang, and Specia 2016; Specia, Frank, Sima’an, and Elliott 2016; Elliott and Kádár 2017; Qian, Zhong, and Zhou 2017). The Third Conference on Machine Translation, which formed part of the Empirical Methods in Natural Language Processing Conference 2018, featured a shared task specifically on multimodal (i.e., image + text) translation: http://statmt.org/wmt18/ multimodal-­task.html. For a discussion about the task, see https://aclweb.org/anthology/ W18-­6402 (last accessed 4 April 2019).

References Díaz Cintas, Jorge, Anna Matamala, and Josélia Neves. 2010. New Insights into Audiovisual Translation and Media Accessibility. Amsterdam and New York: Rodopi. Elliott, Desmond, and Ákos Kádár. 2017. “Imagination Improves Multimodal Translation.” In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 130–41, Taipei, Taiwan. Fryer, Louise. 2016. An introduction to Audio Description: A Practical Guide. London and New York: Routledge.

Concluding remarks  203

Gambier,Yves. 2006.  “Multimodality and Audiovisual Translation.” In Audiovisual Translation Scenarios: Proceedings of the Marie Curie Euroconferences MuTra: Audiovisual Translation Scenarios, Copenhagen 1–5 May, edited by Mary Carroll, Heidrun Gerzymisch-­Arbogast, and Sandra Nauert, Denmark. Jiménez Hurtado, Catalina,Tiina Tuominen, and Anne Ketola, eds. 2018. Methods for the Study of Multimodality in Translation. Linguistica Antverpiensia 17. Kaindl, Klaus. 2013. “Multimodality and Translation.” In Routledge Handbook of Translation Studies, edited by Carmen Millán and Francesca Bartrina, 257–69. London and New York: Routledge. Kress, Gunther. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. London and New York: Routledge. Pérez-­González, Luis. 2014a. Audiovisual Translation: Theories, Methods and Issues. London and New York: Routledge. ———. 2014b. “Multimodality in Translation and Interpreting Studies. Theoretical and Methodological Perspectives.” In A Companion to Translation Studies, edited by Sandra Bermann and Catherine Porter, 119–31. Chichester: Wiley-­Blackwell. Qian, Xin, Ziyi Zhong, and Jieli Zhou. 2017. “Multimodal Machine Translation with Reinforcement Learning.” The Computing Research Repository, abs/1805.02356. Remael, Aline, and Nina Reviers. 2018. “Multimodality and Audiovisual Translation: Cohesion in Accessible Films.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-­González, Chapter 17. London and New York: Routledge. Shah, Kashif, Josiah Wang, and Lucia Specia. 2016. “Shef-­multimodal: Grounding Machine Translation on Images.” Proceedings of the First Conference on Machine Translation, 660–65. Snyder, Joel. 2005. “Audio Description: The Visual Made Verbal Across Arts Disciplines Across the Globe.” Translating Today 4: 15–17. Specia, Lucia, Stella Frank, Khalil Sima’an, and Desmond Elliott. 2016. “A Shared Task on Multimodal Machine Translation and Crosslingual Image Description.” Proceedings of the First Conference on Machine Translation, 543–53. Taylor, Christopher. 2004. “Multimodal Text Analysis and Subtitling.” In Perspectives on Multimodality, edited by Eija Ventola, Cassily Charles, and Martin Kaltenbacher, 153–72. Amsterdam: John Benjamins. ———. 2013. “Multimodality and Audiovisual Translation.” In Handbook of Translation Studies, edited by Yves Gambier and Luc van Doorslaer, Vol. 4, 98–104. Amsterdam: John Benjamins. ———. 2016. “The Multimodal Approach in Audiovisual Translation.” In Audiovisual Translation Theoretical and Methodological Challenges, edited by Yves Gambier and Sara Ramos Pinto. Special issue of Target 28 (2): 222–36. Tymoczko, Maria. 2005. “Trajectories of Research in Translation Studies.” Meta 50 (4): 1082–97.

INDEX

accented cinema 101; see also diaspora film; Naficy, Hamid action theory 54 activist subtitling see subtitling Adami, Elisabetta 11, 17 – 18, 55, 95, 199 – 200 adaptation see translation advertising 4 – 5, 17, 56, 80, 83, 85, 201 affinity spaces 105 affordances 4, 13, 15 – 16, 26, 28 – 33, 37 – 8, 45, 81, 88, 96, 104 – 8, 110, 137, 139, 145, 150 Albright, Daniel 159, 162 alif 127 anime 73 – 4, 77, 82, 86, 90n1, 104, 107 anthropology 9, 89 Apollinaire, Guillaume 19, 140 – 53 Arabic see language artificial intelligence see technology assistive mediation 201 audiovisual translation see translation ballet 3 – 4, 19, 140, 162 – 75 Bannerman, Henrietta 163, 174 Bassnett, Susan 6 – 10, 53, 160 Beharry, Shauna 193 Berger, John 181 Berman, Antoine 122 Bezemer, Jeff 5, 12 – 13 “black” music see music Blake, William 135, 145 blindness 187 – 8, 190 blues, the see music Boethius 2

Bracquemond, Félix 146 Browning, Robert 123 Burnard, Pamela 160, 164 captioning see digital culture Carroll, Lewis 4 censorship 10 children’s literature 4, 9, 54, 82, 135, 201 China 107, 111, 187 Chinese Whispers 24 – 7 choreomusicology 158 – 66, 170 cinema of attractions 98, 103 Cocteau, Jean 173 – 4 code see semiotics coherence 14, 32 – 3, 74 Cold War, the 7, 61 comics 10, 16, 71, 78, 82, 136, 201 Commedia 18, 119 – 20, 124 – 5, 128 – 9 communication: channels of 12, 51, 53; codification of 75 – 7, 85, 90; communicative aim 55, 72, 100, 108; in films 96 – 7, 102; intercultural 11, 15, 71 – 2, 75, 89, 104, 160 – 1, 164; intermodal 16, 58, 78 (see also multimodality); interpersonal 28, 80, 102 – 3; and language (see language); limits of 136; in literature 117, 124, 134 – 6; materiality of 95, 104; and meaning 1, 3 – 5, 12, 28, 160 (see also meaning); monomodal 11, 14, 94 (see also monomodality); multimodal 3, 5, 11 – 13, 30, 37, 45, 51 – 6, 59, 64, 77, 110, 135 – 6, 142, 160, 163, 198 – 9 (see also multimodality); nonverbal (see modes);

Index  205

purpose of 51; and social interaction 12 – 13, 27 – 8, 31 – 2, 36, 45, 51, 71, 75, 103 – 4, 201; theories of 25, 36, 51, 59 – 61; and translation 1 (see also translation); visual 89 computer games see technology context: communicative 57; cultural 2, 5 – 6, 9 – 10, 15, 59, 55, 62, 64, 72; disciplinary 6; linguistic 1, 10; multimodal 3, 5 (see also multimodality); nonverbal 71; re-contextualisation 38; social 8, 12, 50 – 2, 55, 62, 64; temporal, medial 50 – 1, 58; mediation 57 co-text see text types crowdsourcing 10, 111 cultural mediation 199 culture: digital (see digital culture); and globalisation 4; and language (see language); and modes 135, 140; and multimodality 72 – 5; national 2, 29, 88; and semiotics (see social semiotics); and thought 118; and translation 6, 8 – 12, 15, 51 – 5, 76 – 7, 97, 160 – 3, 166, 172 – 5; see also context dance 3, 6, 19, 28, 31, 63, 121, 154, 158 – 75, 200 – 1 danmu 18, 96, 106 – 11, 200 Dante: the character and the man 117; De vulgari eloquentia 118, 120, 125, 129; original language 117 – 19, 126 – 7; and translation 18, 117 – 31, 153; see also Commedia Debussy, Claude 3, 16, 19, 166 – 9 denotation 74, 136, 142, 154 Derrida, Jacques 57, 118, 122 – 3, 128, 136, 140, 180 – 1, 188 – 91 de Saussure, Ferdinand 13 Diaghilev, Serge 3, 166, 173 dialects see language diaspora film 193 Dickens, Charles 9, 135 diegesis 81 – 2, 96 – 111 digital culture: cameras 57, 74, 100, 146, 188; captioning 4, 102 – 3, 106; and corpora 87, 200; democratic nature of 18, 95 – 6, 105; e-literature 4, 138 (see also poetry); emojis 4; hypertextual fiction 201; memes 4, 11, 80, 85; participatory media 103 – 11, 200; social media 4 – 5, 11, 103, 106; and subtitling (see subtitling); virtual communities 104, 109 digital technologies see technology digital texts see text types digressions 179 – 80 Dionisi, Giovanni Iacopo 126 – 7, 129

discourse analysis 5, 56 Divine Comedy, the see Commedia domesticating translations see translation studies drawing 25, 34 – 5, 38, 42, 47, 179 – 97 Dryden, John 122 – 3, 136 dubbing see subtitling Dufy, Raoul 19, 144, 146, 150 – 1 Dürer, Albrecht 146 Eastwood, Clint 141 Eco, Umberto 52, 195 ekphrasis 6, 13, 139 e-literature see digital culture emojis see digital culture ensembles: audiovisual 111; cultural context of 18, 96, 138; decodification of 96; definition of 13, 135; dimensionality of 138 – 9, 142 – 4, 148 – 9, 152, 154; and films 96; and machine translation 201; and meaning 32 – 3, 38 – 9, 201; as multimodal complexes 11 – 13, 18 – 19, 33 – 4, 46, 73, 79, 135 – 8, 147 – 55, 200; orchestration of 14, 135, 148, 152; as texts 73; translation of 19, 38 – 9, 74, 83 – 5, 135 – 8, 142, 147 – 54, 200 – 1 (see also translation) epistemology 31, 37 – 8, 45, 74, 77, 86 equivalence see translation studies ethics 8, 10, 106, 200 ethnocentrism 9 eye(s) 87, 97 – 9, 137, 168, 170, 179, 181 – 5, 188 – 95 faithfulness see translations studies fansubbing see subtitling feminism 6, 9 – 10, 191; see also translation studies feminist translation see translation studies film sound 95 – 6, 98 – 101 film studies 51, 82, 98 Foucault, Michel 52, 147, 150 French see language Gauguin, Paul 146 gender 2, 6, 10, 53, 56 Genette, Gérard 6, 102 genre: and culture 57, 60 – 2, 64 – 5; definition of 57 – 8, 65, 199; in film 97, 102, 104; in font 141; and language 120; of multimodal texts 6, 14, 39, 43, 82 – 3; in music 60 – 3; and national codification and circulation 85 – 7; and plurality 192; and translation 32, 58 – 9, 63 – 5, 84, 148, 162, 172 German see language

206 Index

gesture: artistic 175; and dance 169 – 70, 172; and equivalence 163 – 5, 170; gestural analysis 71; gestural arts 164 – 70; gestural interplay; and meaning 74 – 6, 80, 88, 161 (see also modes); musical 161 – 3; nongestural 5; physical 159 – 60, 163; and sound 171 globalisation 3 – 4, 10 – 11, 71, 95, 173 Goodman, Nelson 121 Gorbman, Claudia 168, 175 Halliday, Michael 11 – 13, 28, 32, 72 haptic visuality 193 – 4 Hazlitt, William 134 – 5 Holmes, James 8 Humphrey, Doris 170 – 1 hypertexts see texts types iconology 6 ideology 6, 53, 99 – 100 image see modes innovation see technology intelligibility 180 interculturality 11, 15, 71 – 2, 75, 89, 104, 160 – 4, 166, 172, 175 interdisciplinarity 11, 51, 64 – 5, 161, 201 – 2 intermediality 50, 56, 59 – 60, 64 intermodality see translation interpreting see translation intersemiotic translation see translation intertitles see subtitling Italian see language Jakobson, Roman 7, 31, 46, 49, 53, 58 – 60, 65, 65n1 Jewitt, Carey 5, 12 – 13, 52, 54 – 6 Jordan, Stephanie 158 – 9, 170 – 2 Joyce, James 153 – 4 Kaindl, Klaus 7, 15 – 18, 39, 198 – 200 Kloepfer, Rolf 51 Kosminsky, Peter 101 Kress, Gunther 5 – 7, 13 – 18, 50 – 2, 54 – 6, 59, 78, 86, 96, 99, 106, 118 – 19, 136, 139 – 40, 142, 160 – 1, 165, 198 – 9, 201 Kuhn, Thomas 50 Lacoue-Labarthe, Philippe 192 language: Arabic 82, 101, 126 – 7; and the arts 124, 159; body language 121; Chinese 4, 110; and communication 12, 24, 27 – 9, 75, 199; and complexity theory 86; and cultural context 2, 6, 12, 15, 71 – 4, 153 – 4; dialects 109, 119 – 20; French 1 – 3, 6, 14 – 16, 60, 123 – 4, 128 – 9, 139, 143 – 4, 149, 151,

160, 180; German 3, 14, 18, 57 – 8, 60 – 1, 63, 122, 128 – 9, 139; Italian 119 – 20, 125 – 8, 130 – 1, 153, 139; language change 120 – 1; Latin 14 – 16, 120, 122 – 3, 126 – 8; lexis 15, 31, 140, 143; and meaning making 119, 130 – 1; metakinetic 163; metalinguistic 163; as a mode 14, 16 – 17, 65, 122, 199 (see also modes); multilingualism 11, 75 – 7, 86, 119, 160; and multimodality (see multimodality); and neuroscience 89; original language 196n5 (see also Dante); relation to the world 72 – 3; register 12, 74, 119 – 20; and relevance theory 86; as a semiotic exception 76, 84; Spanish 3, 129, 139; and speech (see speech); standardisation of 75 – 6, 83, 90, 123; style 74, 108, 119, 148; textual 159 – 60, 164; and translation (see translation); verbal 78, 87 – 8, 127, 159 – 60, 163 – 4; writing (see writing) Latin see language Lefevere, André 6, 8, 53 – 4 lexis see language linguistics 7, 9, 29, 44, 53, 73, 88 – 90, 118 linguistic standardisation see language literary criticism 5, 135, 137, 141 – 2, 145 livres d’artistes 19, 96, 136, 144, 146 localisation see translation logocentrism 14, 78, 142, 147, 160 machine translation see technology Mallarmé, Stéphane 3, 16, 166 – 9 Marks, Laura U. 191, 193 Martinet, André 7 meaning: and agency 28, 38, 54, 72, 90, 96, 201; communication of 1, 3 – 5, 12, 14, 52; (re)constitution of 27, 35, 37 – 9, 41 – 4, 57, 136, 139 – 40, 143, 145, 147 – 8, 154, 198 – 9; construction of 10, 12 – 13, 24 – 47, 51 – 2, 59, 160, 165 – 6, 168, 173 – 5; forms of meaning making 2, 5, 12, 14, 18, 34, 55, 166, 199 – 200; intergeneric 60 – 2; intermedial (see intermediality); intermodal 5, 15 – 16, 58, 60, 78, 81, 87, 96, 111, 139, 148, 151 – 2, 199; intramodal 6, 15 – 16, 60, 142 – 4; meaning of 1; and metaphor 30, 34 (see also metaphor); and multimodality (see multimodality); potential for 24, 26, 28 – 9, 30, 32 – 3, 38, 57 – 8, 72, 74, 76, 106, 141; resources for 14, 17, 26 – 34, 37, 39, 41, 45 – 6, 51 – 2, 56, 71 – 90, 94, 96, 111, 128, 142, 148 (see also modes); transduction of 5, 15 – 16, 26, 37, 46, 59, 142, 151, 165, 199; transference

Index  207

of 3, 5 – 6, 8, 19, 53 – 65, 165, 175, 198; transformation of 5, 15, 19, 36, 42, 45 – 6, 59, 64, 94 – 5, 142, 165; translation of (see translation); transmutation of 58, 60; transposition of 8, 17, 27 – 47, 59, 136, 142, 147 – 8, 150, 154, 172, 198 – 9 medium 18, 50 – 1, 56 – 65, 85 – 7, 98, 118, 161, 163, 170 – 1, 199; see also translation memes see digital culture metaphor 15, 28, 30, 34, 42, 44, 46, 86, 122, 140, 158 – 9, 163, 170 – 2, 184, 198 metre see poetry Minh-ha, Trinh 100, 103 Minors, Helen Julia 6, 19, 47, 55, 78, 140, 198, 200 Mitchell, W. J. T. 121 mobile phones see technology modes: affordances of (see affordances); blending of 163, 167 – 8; dance as 3, 6, 19, 28, 31, 159 – 61, 85, 168 – 75 (see also dance); definition of 12 – 14, 19, 29, 118 – 19, 137 – 40, 161, 199; development of 29, 31 – 2, 86; disaggregation of 144, 147, 150, 154; and discourse 12 – 13, 50 – 2, 55, 58, 159 (see also discourse analysis); distal 137, 139 – 40, 151; drawing as 179 – 97 (see also drawing); ensembles of (see ensembles); fonts 109, 137, 139 – 42; image as 13 – 15, 25 – 7, 31, 33, 37, 41, 47, 71 – 90, 128, 142, 145, 148, 152; latent 137, 151; language as (see language); music as (see music); non-gestural (see gesture); non-linguistic 17 – 18, 54, 121; nonverbal 4 – 5, 55, 72 – 3, 76, 78 – 84, 87 – 90; not countable 119; ontology of 16, 37 – 8, 45 – 6, 95 – 6, 136, 138, 142, 154, 199; and partiality 27, 32 – 3, 35 – 6, 185; prioritising in translation 145, 147 – 52, 154; proximal 137, 139, 144, 150 – 1, 153 – 4; relationships between 16, 51, 58, 78, 81 – 2, 87, 121, 137 – 9, 145, 147, 153, 160, 163 – 5, 170; and semiotic knowledge 75, 77; as semiotic resources 5, 12 – 15, 17, 19, 28 – 32, 34, 41 – 2, 51 – 2, 57, 65, 65n3, 72 – 6, 85, 153, 199, 202; singing as 58, 140, 150 – 2; sound as 4 – 5, 30, 39, 63 – 4, 78, 80, 82, 121; speech as (see speech); submodes 55, 62, 66n9, 108 – 9, 119, 150 – 4; thought as 118; touch as 39, 41 – 2, 181 – 2, 184, 188 – 91, 193; visual 55, 122; writing as (see writing); monomodality 6, 14, 53 – 4, 94, 137, 141, 143, 147 – 8, 201 Morris, Mark 171 – 2 Mounin, George 7

multilingualism see language multimediality 18, 50 – 1, 55 multimodal complexes see ensembles multimodality: and the arts 175; and audiovisual texts 94 – 111; and communication (see communication); and culture (see culture); and digital technology (see digital culture; technology); dimensionality of ensembles (see ensembles); the formal analysis of 142 – 52; framing 36 – 7, 100, 139; and intermodality (see translation); and intramodality (see translation); and language (see language); and linguistics (see linguistics); and literary translation 8 – 9, 18, 122 – 4, 136 – 8, 144 – 5, 148, 153; and multilingualism (see language); multimodal complexes (see ensembles); nonverbal resources (see modes); orchestrating ensembles (see ensembles); processes of design, production, and distribution 51 – 2, 58, 73, 85; social semiotic theories of (see social semiotics); and semiotics (see semiotics); and subtitling (see subtitling); and transdisciplinarity (see transdisciplinarity); transcriptions 44, 46, 165, 200; and transduction 5, 15 – 16, 26, 37, 46, 66n16, 165 (see also meaning); and transformation (see meaning); and translation 24, 71, 75, 83 – 4, 88 – 9, 122 – 31, 180 (see also translation); transmodality 59, 122 Munday, Jeremy 56 music: as accompaniment 150 – 2; arrangement 63, 165; “black” 50, 61 – 2; blues 62; covers 165; jazz 195n2; as a mode 3, 6, 11, 13, 16, 19, 13, 46, 55, 58, 61 – 5, 74 – 7, 88, 118, 135 – 8, 140, 150 – 4, 158 – 75, 200 (see also modes); musicdance works 19, 158, 160 – 6, 169 – 70, 172 – 3, 175, 200; musicking 164; opera 39, 47, 57 – 60, 166, 201; performed vs. written 150 – 2; popular 18, 61; rock ‘n’roll 18, 49 – 50, 60 – 3; Schlager 18, 60 – 5; transcriptions of 46, 165; visualisation of 165, 170 – 2, 174 – 5; “white” 50, 61 – 2 musicology 51, 75, 89, 163 Naficy, Hamid 98, 100 – 1, 103 Nancy, Jean-Luc 179 – 80, 189, 192 neuroscience 89, 121 Newfield, Denise 59 Nida, Eugene 7, 136 Nijinsky,Vaslav 3 – 4, 19, 166 – 70 Norwood, Tamarin 5, 19, 33, 78, 200

208 Index

objectivity 98 – 100, 191 on-screen titles see subtitling ontology 16, 18, 31, 37 – 8, 45 – 6, 95 – 6, 99 – 103, 105 – 11, 136, 138, 142, 154 opera see music original language see Dante page (surface of) 179 – 95 Palumba, Giovanni Battista 146 panaesthetics 162 paratexts see text types Peirce, Charles Sanders 13, 50 pencil: marks on a page 180, 182 – 5, 188; tip of 179 – 83, 186, 188 Pérez-González, Luis 5, 11, 17 – 18, 95–6, 98, 100 – 1, 103 – 4, 106, 109, 200 – 2 performance: audience of 174; and citizenship 18, 96, 200; and drawing 195; of films 99; and meaning construction 160, 162, 164, 171, 175 (see also meaning); multimodal impact of 58; of music 150 – 1; and rhetors 18, 96, 110 – 11; spaces 161; spoken word 122, 159, 201; as text 3, 16 performing arts 158 – 9, 161 phenomenology 152, 191 – 2 Picasso, Pablo 173 plays 54, 122, 136, 201 poetry: adaptation of 3, 5 – 6, 19; animated 153, 201; and ballet 140, 166 – 9 (see also ballet); e-poems 138; and film 100; and fonts 140; and knowledge 193; as literature 1, 118, 134 – 6; metre 60; and multimodality 5, 120, 122, 135 – 6, 139, 169 (see also multimodality); rhyme 2, 6, 14, 137; translation of 1 – 6, 14 – 19, 47, 120 – 4, 140 – 54, 166 – 9 (see also translation);Vietnamese 100 Posner, Roland 51 postcolonialism 9 – 10 post-impressionism 146 post-structuralism 9 Poulenc, Francis 150 – 2 Pousette-Dart, Richard 149 power 6, 10, 39, 52 – 3, 56, 99, 102, 117, 119 Prager, Emily 187 pragmatics 17 Presley, Elvis 18, 61 – 3 psychology 5, 51, 82, 89 publishers 142, 148, 154 Purcell, Henry 171 race 6, 53, 172 Ramos Pinto, Sara 11, 18, 55, 95, 199 – 200 re-constitution of meaning see meaning recording technologies see technology

Rees, Marc 173 – 4 relevance theory 86, 89 Reynolds, Matthew 11, 18, 24, 55, 153, 161, 198 – 9 rhyme see poetry rock ‘n’roll see music Romanticism 7, 58, 135, 192 Romero-Fresco, Pablo 101 Rossetti, Dante Gabriel 1 – 3, 6, 14 – 15 Sakai, Naoki 123 Satie, Erik 19, 173 – 4 Schlager see music Scott, Clive 144, 152 – 3 semiotic resources see modes semiotics: applied 52; arbitrariness of the sign 13, 35, 52, 72 – 3; and code 26, 51, 72, 96, 100, 108 – 9, 127, 131, 174; dynamic nature of 51 – 2, 72, 75 – 7, 84 – 6, 140, 152, 158, 194, ; iconic signs 13, 72, 121; indexical signs 13; intersemiotic translation (see translation); and linguistics 29, 44 (see also linguistics); and multimodality 49 – 50 (see also multimodality); semiosis 12, 34, 45, 59, 86 – 9; signs 13, 28, 30 – 45, 49 – 52, 56, 73 – 90, 108, 121, 124, 127, 130, 201; signifieds 30, 33 – 6, 43 – 5, 72 – 4, 102; signifiers 29 – 36, 38, 41 – 5, 73 – 4, 127; structuralist 51, 72; see also social semiotics sensory planes 159 Shakespeare, William 137 – 8 sight 180 – 95, 121 signifieds see semiotics signifiers see semiotics signs see semiotics singing see modes social agency 51, 54 social media see digital culture social sciences 9, 11 social semiotics: and cultural context 73, 84, 88 – 9; the development of 13, 52, 85, 88; interpersonal 80; levels of meaning in 80; and the making of signs 36, 45; representational 80; and semiotic knowledge 84 (see also semiotics); textual 80; and theories of multimodality 12 – 13, 27, 72 – 4, 87 – 8 (see also multimodality); and translation 27 – 8, 84, 89 – 90, 198 – 9 (see also translation) sociolinguistics 88, 90 sociology 51, 88 – 9 somatic experience 19, 158 – 61, 166 Sorrentino, Paolo 102 sound see modes

Index  209

source 1 – 3, 6 – 8, 19, 53 – 4, 58, 63, 78 – 9, 81, 85, 172, 179 – 80 Spanish see language spatial logic 30, 33, 37, 43, 46, 99, 105 speech 5, 11 – 14, 16, 19, 26 – 33, 37, 39, 41 – 6, 58, 72 – 8, 88, 118 – 20, 126, 138 – 9, 144 – 5, 151 – 4, 161, 169 Sterne, Laurence 4 – 5, 153, 194 Stravinsky, Igor 19, 166, 169, 180 submodes see modes subtitles see subtitling subtitling: aesthetics of 10, 94 – 111; as activism 104 – 6; as an amateur practice 79, 81 – 2, 96, 103 – 11, 200; and censorship 10; and commercialism 79, 96 – 100, 111; as creative practice 101, 104, 111; decotitles 101; diegetic 103; and digital technologies (see digital culture); vs. dubbing 78, 81, 96 – 7; dynamic nature of 83; ethnographic approaches 101; and eye-tracking 97; fansubbing 10, 78, 81 – 2, 104 – 6, 200; functional units of 56; for the hard of hearing 81, 201; development of 18, 79, 96 – 100; industrial 18, 97 – 100; intertitles 10, 96, 103; on-screen titles/text 10, 97, 101 – 2, 106 – 7, 201 (see also danmu); ontology of 94 – 111; and multimodality 5, 56, 78, 81, 94 – 111, 122, 201 (see also multimodality); as professional practice 81, 100, 106; as social semiotic practice (see social semiotics); and sound synchrony/asynchrony 97 – 103, 105, 109; suture 99; and translation 18, 96, 78, 81, 83, 94 – 111 (see also translation) sufism see alif surface(s) 179 – 95 suture subtitling see subtitling synchrony see subtitling synthesising technologies see technology target 1 – 3, 6 – 8, 19, 53, 59, 78, 80, 172, 180 target culture 54, 64 Taylor, Christopher 94 – 5, 100 technology: artificial intelligence 11; and cinema 99 – 101; computer games 4; digital 3 – 4, 10 – 11, 56, 71, 95 – 6, 103 – 7, 110 – 11; human-computer interfaces 11, 109, 201; innovation 3, 11; and literature 138, 144; machine translation 7, 53, 201, 202n2; and medium 56; mobile/smart phones 4, 119; and music 164; recording 3, 44, 63, 96, 99 – 101, 105, 109, 150, 152, 169; and semiotics 47, 106 – 7 (see also semiotics); synthesising 96, 103 – 6, 109 – 11; video games 54, 201

temporal arts 158, 162, 171 – 2, 175 text types: alterity of 9; and context 8, 73 – 90; conventions of 53, 61; co-text 73 – 5, 77, 89; digital 15, 95 – 6, 103 – 11; functional 55; hypertexts 88, 95, 153, 201; monomodal 6 (see also monomodality); multimedial 18, 50 – 1, 55; multimodal 3 – 4, 6, 11, 16, 18 – 19, 55, 59, 64, 71, 73, 75, 78 – 9, 82, 85, 87, 95 – 6, 101, 111 (see also multimodality); paratexts 6; production of 18, 54 – 6; source text (see source); spoken 16, 124 – 5, 137, 139; target text (see target); technical 17, 55, 85; verbal 54, 60, 159 – 60; written 16, 38, 97, 108, 111, 112 – 14, 137 – 8, 141 – 2, 144, 150 – 3, 201 theatre 54, 57 – 8, 123 Thornton, Big Mama 62 Tomalin, Marcus 11, 16, 19, 55, 78, 96, 198 – 9 touch see mode Tower of Babel 119 – 20, 127 – 8 transcriptions see multimodality transdisciplinarity 11, 18, 72, 89, 94 transduction see multimodality transference see meaning transformation see multimodality translation: as an amateur practice 10, 78 – 82, 103 – 11, 200; and adaptation 58 – 9, 64, 79, 165 – 6 (see also poetry; translation studies); artistic 3, 94, 161, 164 – 6, 168 – 9; audiovisual 5, 10, 16, 49, 53 – 4, 78, 81, 94 – 5, 100, 162 – 3, 201; context of 8 – 9, 15, 74, 77 – 8; classification of (see translation studies); and cultural identity (see culture); definition of (see translation studies); domesticating approaches to (see translation studies); from drawing to writing 179 – 81; equivalence in 2, 7 – 10 (see also translation studies); the ethnography of 88 – 9, 100 – 1, 200; faithfulness (see translation studies); foreignising approaches to (see translation studies); and gender (see translation studies); impossibility of 3, 195; (inter) cultural 81, 160, 165, 172 – 5 (see also interculturality); interlingual 7, 59 – 60, 65; intermodal 16, 60, 78, 81, 87, 139, 148, 151 – 2, 199; interpretative possibilities 3, 38, 45, 125, 158 – 9, 166; and interpreting 38, 56, 66n5, 79 – 83, 122 – 30, 150, 162, 201; intersemiotic 7, 31, 46 – 7, 60, 65, 76, 165; intralingual 7, 46, 59 – 60; intramodal 15, 60, 142 – 4; and language 7 – 10, 16, 18, 29, 31,

210 Index

76 – 8, 81 – 3, 122 – 3 (see also language); and linguistics 73 (see also linguistics); localisation 4, 75, 79 – 83; machine translation (see technology); multimodal (see multimodality); myopic 180 – 1, 189; para-translation 58; and perfection 38, 118, 181, 195; and phenomenology (see phenomenology); post-translation 11, 58; the practice of 1, 4 – 5, 8 – 11, 14, 18, 29 – 31, 36 – 8, 44, 57, 61, 65, 72, 77, 79, 81 – 7, 90, 103 – 11, 122 – 4, 144, 148, 158, 181, 199, 200, 202; as re-writing 127; and modes (see modes); as a professional practice 81 – 2, 84; representation of 123 – 4; and semiotics 72, 76, 88 – 9 (see also semiotics); sensory 165 – 70, 175; source (see source); and subtitling (see subtitling); transmission of 95, 125; transmodal (see multimodality); transportation/transposition of meaning (see meaning); theories of (see translation studies) translation studies: action-oriented 59; and adaptation studies 64; classification of 6 – 9, 14; definition of 7 – 8, 24, 31 – 2, 39, 42, 45, 47, 77 – 8; descriptive 59; development of 3, 6 – 17, 49 – 58, 71, 79, 152, 201; domesticating translations 6, 9, 90, 146; faithfulness 8, 36, 83, 188, 194; feminist theories 10; foreignising translations 2, 6, 64, 90n6; formal and dynamic equivalence 7, 53 – 4, 58, 73, 75 – 6, 78 – 9, 82 – 3; and language 18,

57, 59, 163; and multimodality 16 – 18, 49, 53 – 8, 60, 64 – 5, 94, 163, 198, 200, 202 (see also multimodality); and music 161 – 3 (see also music); post-translation studies 11; sociological 6, 52 – 3, 59; the translation game 24 – 6; translator training 8 – 9, 65, 72, 77, 81, 84 – 5, 122, 200 translator training see translation studies transmediality 59 transmutation see meaning transnationality 11, 18, 71 – 2, 75 – 7, 83 – 6 transposition see meaning transsensorial perception 169 Ungaretti, Giuseppe 15 van Leeuwen, Theo 5, 13, 50 – 2, 54 – 6, 96, 99, 105 – 6 Venuti, Lawrence 6, 9, 136, 160, 180 video games see technology Villon, François 1 – 2, 6, 14 – 15 virtual communities see digital culture visualisation see music von Flotow, Luise 6, 10 Wei, Wang 3 “white” music see music woodcuts 19, 135, 144 – 6, 148, 152 writing 2, 5, 14, 16, 19, 26 – 33, 37 – 42, 46 – 7, 71 – 8, 88, 119 – 20, 126, 136, 139, 141 – 7, 153 – 4, 179 – 97 Zukofsky, Celia and Louis 123