Film Text Analysis: New Perspectives on the Analysis of Filmic Meaning 9781138911383, 9781315692746

Table of contents :
Cover
Half Title
Title Page
Copyright Page
Contents
List of Table and Figures
Preface and Acknowledgments
1 Introduction: Bringing Together New Perspectives of Film Text Analysis
2 Towards a Semiotics of Film lighting
3 Editing Space as an Audio-Visual Composition
4 Movie Physics or Dynamic Patterns as the Skeleton of Movies
5 From Visual Narrative Grammar to Filmic Narrative Grammar: The Narrative Structure of Static and Moving Images
6 From Text to Recipient: Pragmatic Insights for Filmic Meaning Construction
7 Intermediality in Film: A Blending-Based Perspective
8 EAT, PRAY, LOVE: Expanding Adaptations and Global Tourism
9 Conclusion: Film Text Analysis – A New Beginning?
List of Contributors
Film Index
Name Index
Subject Index

Citation preview

Film Text Analysis

“The first collection to give an overview of the field of film text analysis including a wide variety of approaches, methodological advances and in-depth analyses hinting at the possibilities of multimodality studies for film.” —Florian Mundhenke, University of Leipzig, Germany This book examines film as a multimodal text and an audiovisual synthesis, bringing together current work within the fields of narratology, philosophy, multimodal analysis, sound as well as cultural studies in order to cover a wide range of international academic interest. The book provides new insights into current work and turns the discussion towards recent research questions and analyses, representing and constituting in each contribution new work in the discipline of film text analysis. With the help of various example analyses, all showing the methodological applicability of the discussed issues, the collection provides novel ways of considering film as one of the most complex and at the same time broadly comprehensible texts. Janina Wildfeuer is a Researcher in Multimodal Linguistics in the Linguistics Department of the University of Bremen, Germany, specializing in multimodal linguistics and media studies. Her recent publications include a monograph on Film Discourse Interpretation from 2014 as well as an edited collection of papers building bridges for multimodal research (2015). John A. Bateman is a Full Professor of Applied Linguistics in the English and Linguistics Departments of the University of Bremen, Germany, specializing in functional, computational and multimodal linguistics. His recent publications include monographs on Multimodal Film Analysis (2012) as well as on the Text and Image divide (2014).

Routledge Advances in Film Studies For a full list of titles in this series, please visit www.routledge.com.

41 The Western in the Global South Edited by MaryEllen Higgins, Rita Keresztesi, and Dayna Oscherwitz 42 Spaces of the Cinematic Home Behind the Screen Door Edited by Eleanor Andrews, Stella Hockenhull, and Fran Pheasant-Kelly 43 Spectacle in “Classical” Cinemas Musicality and Historicity in the 1930s Tom Brown 44 Rashomon Effects Kurosawa, Rashomon, and Their Legacies Edited by Blair Davis, Robert Anderson and Jan Walls 45 Mobility and Migration in Film and Moving Image Art Cinema Beyond Europe Nilgün Bayraktar 46 The Other in Contemporary Migrant Cinema Imagining a New Europe? Guido Rings 47 Horror Film and Affect Towards a Corporeal Model of Viewership Xavier Aldana Reyes 48 India’s New Independent Cinema Rise of the Hybrid Ashvin Immanuel Devasundaram 49 Early Race Filmmaking in America Edited by Barbara Tepa Lupack 50 Film Text Analysis New Perspectives on the Analysis of Filmic Meaning Edited by Janina Wildfeuer and John A. Bateman

Film Text Analysis New Perspectives on the Analysis of Filmic Meaning

Edited by Janina Wildfeuer and John A. Bateman

First published 2017 by Routledge 711 Third Avenue, New York, NY 10017 and by Routledge 2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN Routledge is an imprint of the Taylor & Francis Group, an informa business © 2017 Taylor & Francis The right of the editors to be identified as the authors of the editorial material, and of the authors for their individual chapters, has been asserted in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988. All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. Trademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Library of Congress Cataloging-in-Publication Data Names: Wildfeuer, Janina, 1984– editor. | Bateman, John A., editor. Title: Film text analysis: new perspectives on the analysis of filmic meaning / edited by Janina Wildfeuer and John A. Bateman. Description: New York: Routledge, [2016] | Series: Routledge advances in film studies | Includes bibliographical references and index. Identifiers: LCCN 2016022280 Subjects: LCSH: Motion pictures—Philosophy. | Motion pictures—Semiotics. Classification: LCC PN1995 .F4666 2016 | DDC 791.4301—dc23 LC record available at https://lccn.loc.gov/2016022280 ISBN: 978-1-138-91138-3 (hbk) ISBN: 978-1-315-69274-6 (ebk) Typeset in Sabon by codeMantra

Contents

List of Table and Figures Preface and Acknowledgments 1

Introduction: Bringing Together New Perspectives of Film Text Analysis

vii xi

1

J O H N A . BAT E MAN & JAN I NA WI L DF E UE R

2

Towards a Semiotics of Film Lighting

24

T H E O VA N L E E UWE N & MO RTE N B O E RI I S

3

Editing Space as an Audio-Visual Composition

46

M A RT I N E H U VE N N E

4

Movie Physics or Dynamic Patterns as the Skeleton of Movies

66

WO L F G A N G WI L DGE N

5

From Visual Narrative Grammar to Filmic Narrative Grammar: The Narrative Structure of Static and Moving Images

94

NEIL COHN

6

From Text to Recipient: Pragmatic Insights for Filmic Meaning Construction

118

JA N I NA WI L D F E UE R

7

Intermediality in Film: A Blending-Based Perspective

141

J O H N A . BAT E MAN

8

Eat, Pray, LovE: Expanding Adaptations and Global Tourism J OY C E GO G G IN

169

vi

Contents

9

Conclusion: Film Text Analysis – A New Beginning?

187

JA N I NA WI L D F E UE R & JO H N A. BATE MAN

List of Contributors Film Index Name Index Subject Index

199 203 205 207

List of Table and Figures

Table 5.1

Gross differences in dimensions between prototypical cases of drawn and filmed narratives

112

Figures 1.1 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.1 3.2 4.1 4.2 4.3 4.4 4.5

A graphical rendition of the ‘crisis’ of textuality and its disciplinary repercussions. Emil Jannings in The Last Laugh (Murnau 1924). Still from Suspicion (Hitchcock 1941). Three-point lighting set up (Young/Petzold 1972: 107). (a) Still from CSI Miami (left) and (b) screen grab from CSI computer game (right). (a) Still from Persona (Bergman 1966) and (b) Still from Kiss me Deadly (Aldrich 1955). The textual meaning potential of lighting. The interpersonal meaning potential of lighting. (a) Still from The Seventh Seal (Bergman 1957) and (b) Still from The Magician (Bergman 1958). The ideational meaning potential of lighting. Graphical illustration of the ‘kinesphere of Ryan’. Graphical illustration of the two universes combined in the scene from Gravity. Pendulums in a series of coupled pendulums and the light trace of a double pendulum. (a) Screenshot from Vertigo (Hitchcock 1958) and (b) Screenshot from North by Northwest (Hitchcock 1959); look into the abyss. (a) Screenshot from Dogville and (b) Screenshot from The Truman Show. (a) Quasi-still in the final scene of Queen Christina with Greta Garbo and (b) pendulum with rest position. Screenshots from the final scene in Bonnie and Clyde (Penn 1967), after many episodes of crime.

7 26 28 29 30 37 38 41 42 43 58 61 69 72 73 77 78

viii  List of Table and Figures 4.6  (a) Belmondo in Breathless (1960) and Dustin Hoffman in The Marathon Man (1976). The camera moves in front of the actors in the street. 79 4.7  (a) Screenshots from Quantum of Solace: place of chase are a road tunnel at Lake Garda and the marble quarries of Carrara; (b) a schematic description of the chase and the archetype of capture below (cf. Wildgen 1982). 84 4.8  Coupled pendulums (the two ropes) and double pendulums (the arm on which the rope hangs also moves). 84 4.9  Screenshots from Quantum of Solace. First row: Fight in the theater foyer (exchange of gunfire in the movie); fight on stage in Bregenz (in the opera Tosca). Tosca stabs Scrapia, the blackmailer. Second row: Parallel fighting in the exploding hotel: Camille against the general – Bond against Greene. 86 5.1  A narrative sequence with two narrative constituents and one subordinate modifying constituent with a Refiner. 98 5.2  Different types of narrative conjunction using the repetition of a single narrative category (Initials) to show various semantic information (actions, characters in a scene, parts of an individual, or semantically associated elements), which could also be framed by a single image. 99 5.3  Narrative grammar applied to a sequence from Star Wars (12:00–12:27). 102 5.4  Paraphrase of the narrative grammar for the Star Wars sequence (1977, 12:00–12:27) in Figure 5.3. 103 5.5 Polymorphic divisional panel of a bee flying. 110 6.1  The levels of filmic text and discourse (following a more general description of text and discourse in Wildfeuer forthcoming). 126 6.2  Inserts, title sequence, and cut to long shot in Gravity. The inserts in the first shot read “At 600km above the planet Earth the temperature fluctuates between +258 and –148 degrees Fahrenheit. There is nothing to carry sound. No air pressure. No oxygen. Life in space is impossible.” 128 6.3  Two frames from the very long shot at the beginning of Gravity, showing a white object becoming visible. 132 7.1  The classic blending example: the respective semantics of the words ‘boathouse’ (left) and ‘houseboat’ (right) generated by two contrasting blending diagrams operating over the same ‘input spaces’. 146 7.2  Intermedial citation of Leonardo da Vinci’s ‘Last Supper’ (shown right) in Robert Altman’s Mash (1970, shown left) together with correspondence ‘mappings’ with the original. 147

List of Table and Figures Running the blend to derive further interpretations of details from the input spaces, such as a ‘lamp ~ halo’ association and blended transfer of spiritual attributes. 7.4 A frame sequence from the ‘Robin Hood’ scene from Andrew Adamson and Vicky Jenson’s Shrek (2001, at approximately 0:50:00). 7.5 Representative frames from the opening sequence of Paul Verhoeven’s Starship Troopers (1997). 7.6 Frames from the congressional hearing scene from Jon Fravreau’s Iron Man 2 (2010). 7.7 Representative frames from the opening sequence of Doug Liman’s Edge Of Tomorrow (2014). 7.8 Representative frames from the production company logo sequence at the beginning of Liman’s Edge Of Tomorrow (2014). 7.9 Suggestive blend diagram involving the mediated communicative situations of film and live broadcast news. 7.10 Opening sequence from Ron Howard’s Frost/Nixon (2008). 8.1 Julia Roberts emotes in front of cardboard cut-out locals in Eat, Pray, Love. 8.2 Julia Roberts with one of her squantos in Eat, Pray, Love.

ix

7.3

149 151 154 156 157 158 159 161 180 181

This page intentionally left blank

Preface and Acknowledgments

The original idea for this book emerged from a lecture series entitled “Recent Paradigms of Film Studies”, which took place at the University of Bremen, Germany, in summer 2014, organized by the Bremen Institute for Transmedial Textuality Research (BItT). The BItT examines various forms of cross-media textuality in its inter-, intra-, and transmedial instantiations and is interested in the analysis, interpretation, and evaluation of factual and fictional texts in various media. A particular interest lies in the analysis of filmic texts as one of the most powerful audio-visual narratives, which are today approached from a myriad of perspectives and research areas. The lecture series was therefore intended to bring together a variety of innovative research issues in the broad context of film studies, inviting contributions from disciplines such as narratology, philosophy, multimodal analysis, sound as well as cultural studies. By concentrating on film as a particular class of text, we assume that this artifact has its own logical form of meaning production that can and should be analyzed by capturing the notion of textuality. For the lecture series, we particularly asked for new developments and progress made in our own disciplines and those affiliated to the institute as well as in other disciplines connected to the analysis of filmic text. We thank all contributors to the lecture series for their interest in our institutional work, their presentations in the lecture series, and the lively conversations and discussions we had during the summer semester 2014. We also thank the institute for the financial support of the lecture series, which made it possible to bring distinguished film scholars to Bremen. The book takes the intention of the lecture series as its main starting point in order to make the talks in this lecture series, their theoretical, methodological, and analytical approaches, as well as our own thoughts on this topic available to a broader audience. It thus builds a shared context for the existing diversity among film analytical approaches and at the same time opens the field for new and innovative research. We thank all contributors to this volume for their continuous cooperation and intellectual and creative support during the processes of writing, editing, and revising the book. It was a pleasure to work with all of you! Janina Wildfeuer and John A. Bateman Bremen, August 2016

This page intentionally left blank

1

Introduction Bringing Together New Perspectives of Film Text Analysis John A. Bateman & Janina Wildfeuer

1 Introduction to the Introduction Just as the contributions to this collection were being put into their final form, and the first versions of this introduction were being drafted, there was a happy coincidence in the selection of topic in Richard Dyer’s 2016 contribution to the Kracauer Lectures in Film and Media Theory at the Goethe-University Frankfurt (Dyer 2016; see also the reference in Wildfeuer in this volume). Dyer’s discussion of “The Persistence of Textual Analysis” articulated many of the points giving rise to and motivating the direction that we pursue here. The idea of ‘text’ or ‘textual’ analysis in film is at the same time one that is traditional, often to the point of being considered old fashioned, and one that remains subject to ongoing critique. Several of the themes taken up by Dyer resonate closely with the direction in which we wish to suggest pushing film text analysis, both with respect to his considerations of criticisms brought against the notion of ‘text’ in film and his proposals for taking film text analysis further. Dyer begins with the idea that film text analysis is “nothing special” – which is a reoccurring motif among those talking of ‘textual analysis’ nowadays. Under this view, it is simply a skill that is grounded in “looking and listening” closely to what is actually (physically, manifestly) present in individual films and film extracts. In terms reminiscent of a further discussion from David Bordwell, such looking and listening must also be centrally reliant on ‘common sense’ (Bordwell 2011). Dyer’s view of ‘filmic text’ and its analysis then focuses on starting from the image and sound in order to pull out the range of meanings, affects, and emotions that a film or film extract ‘makes available’ for its audience or spectators. In many contexts such an approach is so naturalized as to be indistinguishable from analyzing film as such – after all, each film analysis starts with considering what the films and film segments analyzed may mean for their spectators. Variations (and critiques) surface when the next step is taken, i.e., when it is asked just how such potentials for meanings might be found. Analytic attention then turns to assessing the role and importance of the films’ contexts of production, of their contexts of consumption (audience studies), and of the contexts of cultural knowledge employed when engaging with any ‘text’. Dyer wishes to argue that, regardless of any such extensions and broadening of attention, engagement with actual film material should be

2

John A. Bateman & Janina Wildfeuer

maintained as a constitutive and definitional component of film studies. And, as we shall see below, this is indeed a powerful strategy for avoiding and defusing some of the problems that arise from many of the less restrained notions of what might constitute a ‘text’ that have emerged over the past half-century. The principal and inescapable question remaining is just how that is to be done in a manner that is not only appropriate for film in all the medium’s multimodal richness, but which is also systematic, precise, and rigorous (to any of the standards applicable to ‘humanistic’ knowledge of this kind). This emphasis places the search for suitable methodologies that support more precise practices of film text analysis in a particularly central position, in order to avoid more subjective, individual ‘readings’ where one view is ‘as good’ as any other. For reasons that we will summarize below, the predominant methodologies currently employed within film studies have come to draw overwhelmingly on literary standards – and this despite one of the critiques of the applicability of ‘text’ in film analysis that Dyer mentions, i.e., that ‘text’ is traditionally very closely bound to the products of (literary) writing. Any such grounding of ‘text’ among the concerns of literature demands that the inherent relationship arising as a consequence between ‘text’ and ‘film’ be theoretically problematized. After all, it takes several steps in abstraction to bring ‘written literature’ and ‘film’ together under a single analytic category in any intelligible fashion; it is by no means considered obvious by all that such steps are worthwhile, or even possible. The development of a path of abstraction by which ‘written text’ and ‘film’ come to be closely related has been supported by the increasingly flexible use made of the notion of ‘text’ itself in literary studies, in cultural studies, and in almost all branches of semiotics. But this very generality brings its own raft of problems. For many, Dyer included, ‘textual analysis’ and ‘analysis’ come to be treated as more or less synonymous – a certain behavior or artifact, regardless of appropriateness, may be described as ‘text’, or ‘textual’, simply as a by-product of conducting the analysis. For others, ‘text’ as such is a domain of contention and struggle, marking both strong ideological orientations and deep divisions within and across disciplinary discourses (cf. Mowitt 1992). Questions of the cultural and ideological specificity of ‘readings’ of texts, of the ‘boundaries’ (if any) of texts, or of the extent to which ‘texts’ are defined as (and by virtue of) including their interpretations have been discussed in detail. These issues have all worked against the achievement of methodologies capable of supporting textual analysis in its ‘original’ (which we will set out more below) sense of engaging closely with material manifestations. Indeed, the notion of text has often become so diluted and general that it is impossible for the specific textual qualities of individual types of text (e.g., film) to be respected sufficiently for reliable analysis. Much of what is specific to ‘texts’ and their ‘textuality’ then falls out of focus, leading to justified critiques of the relevance and appropriateness of ‘text’ analytic approaches tout court.

Introduction

3

This has proved particularly divisive in film studies, as reflected in the carefully patrolled borders still being drawn between literary-culturally inflected film studies, film aesthetics, and approaches building on, or with, the psychology and neuroscience of film. The earlier, more analytic and closely argued textual analyses of individual films that emerged primarily in the 1960s and 1970s (see below) have, over the years, been reconfigured in terms of more impressionistic, distanced readings of film as ‘cultural texts’, what Bordwell (1989) has discussed in terms of ‘symptomatic interpretations’. Some notion of ‘text’ remains over the course of this transitioning, but much of the methodology and precision of those earlier accounts, even if quite limited or more prospective than actual, does not. Indeed, if they appeared today, those very accounts might well face criticism as reductionist, exhibiting insufficient sensitivity to the aesthetics of the medium, lacking attention to historical context or conditions of production, and so on. On the one hand, such critiques are appropriate and necessary because they increase the self-critical reflexivity necessary for analysis of any kind; on the other, they have equally often undermined much of what made those earlier text analyses compelling – i.e., their search and reliance on patterns in the material that is being examined. This aim, which we will take as the hallmark of text analysis, requires far stronger methodological underpinnings to be pursued, methodological underpinnings that current disciplinary borders have made it difficult to articulate. Much of what we undertake in this collection of articles illustrating different forms of textual analysis directly addresses this methodological difficulty. Our claim is that the primary consequence of diluting the notion of ‘text’ to serve a variety of disciplinary masters has been to compromise precisely those benefits that accrue when more finely articulated notions of ‘textuality’ are allowed to serve as methodological guides for the theory and practice of film analysis. This means that we will be working throughout the book with a more restricted notion of ‘text’, one that is not only capable of regaining the precision of the individual, detailed analysis of particulars seen in early film text work, but which is also responsive to the many advances that have been made since then concerning our understandings of what texts are and how they work as mediators between material artifact and interpretation/response. To return, then, once more to Dyer’s assessment of the state of film text analysis: we agree both that it is necessary to orient to the ‘text’, the concrete film material supporting an analysis, and that, for this to be anything more than a subjective (if informed) response to what strikes one as ‘significant’ (or not) in the work under study, a stronger methodological construction of what constitutes ‘text’ and its study is essential. This is the ‘new perspectives’ intended in our title: there are understandings of text and textuality that have, as we shall see, moved on considerably from what is commonly addressed in film studies. And these understandings undercut most of the critiques still being brought against the idea of ‘text analysis’ when applied

4

John A. Bateman & Janina Wildfeuer

to film. Developments of this kind open up an important area for both discussion and practical work, particularly when we note that Dyer’s call for methodological rigor in his lecture actually remained just that, a call. Suitable methodologies for guiding film analysis beyond simply ‘looking and listening’, even if done with considerable attentiveness, sensitivity, and knowledge, were not forthcoming. This gap is then precisely what the present edited collection of papers responds to. To prepare the way for the illustrations of method that our contributions present, however, it is necessary to remind ourselves of the long and complex interaction between ‘text’ and film theory that has shaped, and which continues to shape, the field. This background is important to our purpose because presuppositions and judgments concerning both the nature of text and its applicability to film have become as diverse as theories of ‘text’ itself – and these presuppositions and judgments are by no means always supportive of analysis. A short review of the emergence of ‘film text analysis’ within film studies will consequently offer an appropriate contextualization, assist in moving us towards the new perspective on ‘film as text’ that we pursue, and finally, situate that view within a more contemporary constellation of understandings of what ‘text’ can be. This will, in turn, let us approach the diverse analyses offered in the following chapters in a way that brings out more readily their many points of interconnections and common directions for the future, as well as showing their natural differences in orientation and questions posed.

2 The Notion of Filmic Text in Film Theory Despite reoccurring criticisms, the broad notion of ‘film as text’ has of course accompanied film since the medium’s emergence. Time and again theoreticians and practitioners alike have proposed meaningful ‘text-like’ connections among filmic devices or have discussed similarities between film and language directly. Pudovkin (1926), for example, considered analogies between film images and words. Dziga Vertov, also in the early 1920s, talked of ciné phrase (‘film sentence’) and ciné langue (‘film language-system’). Eisenstein, again from that time, explicitly brought together film and literature, highlighting the ‘organic’ (Eisenstein 1949: 195) relationship across the two by comparing montage structures in both forms. Eisenstein, in fact, drew parallels and similarities in both directions. He argued that the work of the nineteenth-century novelist Charles Dickens was already cinematographic in its organization, while that of the early film-maker David Wark Griffith was equally text-like by virtue of the many specific techniques Griffith introduced for narrating extended stories in film (cf. Wees 1973). By highlighting these narrative or textual functions carried by technical features, such as the use of the close-up “to signify, to give meaning, to designate” as well as montage’s general role as a “means of speaking, a means of communicating ideas” (Eisenstein 1949: 238, 245; emphasis in original), Eisenstein

Introduction

5

did much to prepare the ground for the long and enduring discussion of the relations between film and verbal text that followed. This discussion has grown to be extremely complex, with sites of divergence and contrast echoing not only diverse understandings of the nature of text and language but also equally broad variation in attitudes towards the appropriate methods and institutional role of film studies. We cannot do more than scratch the surface of these debates here and there are, in any case, several extensive and historically grounded characterizations available (cf., e.g., Bordwell 1989). Our point will, therefore, be narrower and oriented more to the future than to a review of the past, although the trajectories set up and which we now attempt to push further are, of course, inescapably products of historical development. This must always also be borne in mind. Contributing to these trajectories are not only disciplinary concerns, including the relative importance given to approaches and questions raised in literary studies, studies of narrative, semiotics, aesthetics, and cognition, but also methodological concerns. These concerns include diverging attitudes adopted to the nature and value of scientific method, to the role of creativity and expressiveness in film and its description, as well as to evaluative judgments concerning the aesthetic appropriateness of filmic techniques – as, for example, with André Bazin’s much discussed rejection of montage in favor of deep focus on the grounds of ‘realism’. While the force of many of these debates has moderated with time, their shaping influences remain, marking out ideological boundaries and disciplinary fracture lines still readily discernible in the paradigms practiced today. We begin by making two of the contributing lines of discussion introduced above more explicit in their own right. The first, theoretically rather more simple, revolved around the issue of whether film could sensibly be considered ‘a language’ at all. Since formulations such as ‘the language of film’ have appeared at least intuitively meaningful, as evidenced by their reoccurrence over the years, it has been necessary to ask on a more theoretical level whether this is just a turn of phrase, a more or less useful metaphor of some kind, or whether it is indicative of a deeper connection between ‘language’ and ‘film’ as communicative practices and/or semiotic systems. Pursuing this, earlier suggestions of more superficial connections, such as for example Pudovkin’s proposals for treating film shots as analogous to words, were made progressively more sophisticated, eventually ending up with the extensive encounters between film theory and semiotics found in the work of Christian Metz, Umberto Eco, Paolo Pasolini, and others (cf. Metz 1966, 1974a, 1974b; Eco 1976; Pasolini 1971). The second line of discussion, considerably more complex, crystallized around the contested notion of ‘text’ itself. The understanding of ‘text’ underwent substantial upheavals in the 1960s, particularly in the context of structuralism and reactions against structuralism – evidenced, for example, in the changing positions of (particularly relevant for film) Roland Barthes (1977). What then might be meant

6

John A. Bateman & Janina Wildfeuer

by taking a film to be a ‘text’ at all became a subject of heated debate. Again, interactions concerning varied positions on this question can be observed across film studies and developments in the traditionally ‘text’-oriented disciplines of both semiotics and linguistics, although the three increasingly went their own ways from the 1970s onwards. Closely entwined and interacting with both lines of discussion have been reoccurring considerations of the nature specifically of film analysis and its appropriate methodologies. Debate, for example, has revolved around whether film analysis should be pursued along the model of the natural sciences, as an ‘empirical-rational’ activity – as already seen with the overtly positivist arguments of filmologie in the 1940s (cf., e.g., Cohen-Séat 1948; Souriau 1951) and now reconfigured in the form of cognitivism and its strong foundations in psychology and the neurosciences (cf., e.g., Bordwell 1989, 2011) – or alternatively draw more on literary styles of interpretation. We see in this latter orientation the full force of approaching ‘text’ as an intrinsically ‘literary’ construct, where the influences exerted by the changing conceptions of ‘text’ from the 1960s and 1970s have had considerable and lasting impact. ‘Text’ then comes to be tracked across a predominantly literary and cultural landscape and constitutes a crucial destabilizing moment which served the valuable task of pushing out the increasingly untenable ‘positivist’ positions of the 1950s and before. Earlier views, according to which texts were to be analyzed ‘scientifically’ for their objective properties so as to allow interpretation to be ‘calculated’, were then replaced in favor of far more fluid views of texts as experienced only in and through acts of ‘reading’. Barthes accordingly defined texts as “methodological fields” (Barthes 1977: 157) for the interaction of diverse, and always ideologically grounded, reading positions; reading thereby becomes so active that it can just as well be portrayed as a form of ‘writing’ (an equally deeply contested term), or re-writing ‘the text’ anew. This orientation is very much alive today and surfaces in discussions alleging texts’ lack/fluidity of boundaries (e.g., how much of a text’s context or, indeed, ‘other’ texts must one include?) as well as in the idea that each interpretation, or reading, creates its own ‘text’. Both positions raise valuable considerations but can also hinder the construction of reliable methodologies if not reined in appropriately. Beneficial distance and perspective can be gained on this situation by reassessing the impact of the upheavals in the understanding of ‘text’ not within particular theoretical perspectives but across disciplines and disciplinary developments. Whereas discussions in individual disciplines are all too easily couched as if they were the only ones to emerge re-configured from their period of reappraisals of the nature of text, this is inaccurate and readily gives rise to disciplinary myopia. In fact, all of the text-oriented disciplines emerged changed from this engagement with the fundamental nature of text, each proceeding onward with respect to  a broadly similar collection of lessons learned but in largely independent ways.

Introduction

7

Figure 1.1 A graphical rendition of the ‘crisis’ of textuality and its disciplinary repercussions.

We suggest this graphically in Figure 1.1 in terms of an outwards flowing reaction to the ‘crisis’ of textuality from the 1960s where distinct disciplinary developments traveled their largely separate trajectories. Despite the tendency just identified of treating other disciplines as if they had not also emerged from the central vortex, in fact, the position and understanding of ‘text’ are now as different in semiotics and linguistics in comparison to former times as it is, for example, in literary studies. All of the disciplines have, therefore ‘moved on’ and yet, as we shall see, this is rarely given sufficient weight in discipline-internal discussions. It is, of course, exceedingly difficult to remain ‘up to date’ in disciplines not one’s own. When making statements that explicitly aim to characterize other disciplines, such restrictions or avoidance of necessary work are no longer justifiable. When making comments concerning other disciplines, those disciplines must be adequately engaged with in their present form, not as one may like to see them for the sake of argument or how they were when one first came into contact with them. To return, however, to the principal starting points of the film text analysis paradigm, we need to pick up the story of the interaction between language and film nearer the center of the vortex.

2.1 From Film as Language … Two points of crystallization for what we will define more narrowly as the beginning of film text analysis are given by the respective activities of Christian Metz and Raymond Bellour; both were working in the highly ‘semiotically-charged’ atmosphere around Roland Barthes in the 1960s where ideas, particularly from Louis Hjelmslev on linguistic method and Roman Jakobson on linguistic poetics and style, were finding application

8

John A. Bateman & Janina Wildfeuer

far beyond their original targets in language. Metz engaged directly with an exploration of a broadly Hjelmslevian semiotics of film as such, while Bellour undertook unprecedentedly detailed analyses of individual films anchored in emerging structuralist analytic techniques. These then form two essential poles for the film text analysis paradigm: attention to material detail on the one hand, and considerations of appropriate methodological and theoretical foundations on the other. We begin with Metz. Hjelmslev’s (1961 [1943]) influential prolegomena had focused specifically on method, on how to address the complex semiotic products of language use, i.e., texts or performances. This was couched in terms sufficiently general as to be picked up by several theorists concerned with issues beyond the narrowly linguistic, including Barthes, Algirdas Julien Greimas, and Metz. Metz consequently began, like many before him, with one foot very much in the camp of objective analyses. His thorough semiotic investigation of the old question as to whether film could be considered a ‘language’ at all (1974a) led to a definition of film as ‘langage sans langue’, i.e., as a system of larger units, without grammatical or syntactical rules (cf. Metz 1974a: 88). Thus, rather than considering filmic organizations to be put together out of determinate and limited ‘syntactic’ material, Metz saw the broader textual ordering of units as responsible for film’s evident intelligibility. As we shall see in a moment, this was already to run at the very edge of semiotic and linguistic theorizing at that time, although not necessarily seen in those terms then. By considering film more aligned with ‘rhetoric’ than ‘grammar’, exploration moved beyond areas for which established methodology could be relied upon. Focusing on textual unfolding was still at that time seen primarily to lie outside of Saussure’s notion of langue, the language system, and so, almost by default, landed back in the realm of performance (in Chomskian terms), parole, and ‘text’ as a unit of language in use (for further discussion of the collapsing of important semiotic distinctions inherent here, see Bateman/Schmidt 2012: 28–42). It was then the textual ordering of material that made it possible, in Metz’s opinion, to maintain space and time, and – consequently – a narrative structure, then famously described in terms of his grande syntagmatique. The grande syntagmatique articulated a model of how larger units of film were manifested as meaningful combinations and sequences of shots, largely independently of their individual (and freely variable) content (cf.  Metz 1974a: 146). Thus, despite their profound theoretical differences, the general structural composition of film already described by Pudovkin and Eisenstein was seen as the primary source for a comparison with language and was placed in the foreground of film’s ability to create coherence in time and space – a criterion later addressed in terms of film’s narrative or textual logic (cf. Bordwell 1989; Bordwell/Thompson 2001). The grande syntagmatique constituted the first, and most significant, filmic semiotic ‘code’ seen by Metz as one of the ways in which an analyst could impose order on the

Introduction

9

material ensembles (following Hjelmslev) of actual films. The relationship between such abstract analytic codes and the work done during production and reception of a film by real film viewers and film makers remained unresolved, requiring as it does the further move from theoretical description to empirical analysis as well as a far more deeply theorized understanding of the relationship between langue and parole. Nevertheless, in subsequent work, such as Metz (1974b), this search for filmic abstract codes was taken considerably further. Within film studies, then, the results of Metz’s explorations provided an extensive theoretical foundation for strongly linguistically influenced approaches to film analysis. In essence, this started both from Hjelmslev’s conception of text as the primary object of study of linguistics (1961 [1943]: 16) and from Hjelmslev’s ‘top-down’ characterization of method as division according to interrelated and interdependent parts. As with several approaches to language, linguistics, and semiotics at that time, the notion of ‘text’ already had an established and central place in Hjelmslev’s thinking and so this orientation was quite natural, although in the end not played out as far as would be necessary to bring the notion of ‘filmic text’ under control. Paying close attention to the ‘textual details’ of the artifacts under investigation was central to this concern and, as we saw above, still resonates strongly in Dyer’s lecture. Noteworthy, however, is that the focus on ‘text’ was maintained in several schools of linguistics and has since resulted in substantial advances, many of which feed into a more contemporary understandings of texts and their mechanisms that we draw on below. These developments were not incorporated in further explorations of ‘filmic texts’, however. It is then reasonable to see Metz’s work as one initiator of systematic textual film analysis (cf. Rhodie 1975), paving the way for its ‘golden period’ in the 1970s and 1980s – as realized in detailed analyses by Bellour (1971, 1974, 1975, 2000), Pasolini (1971), Ropars-Wuilleumier (e.g., 1978, 1981, 1982), Kuntzel (1978), Heath (1981), and Paech (1988), to name just a few. Much of this work adopted the structuralist method handed down from Saussure of seeking systems of contrasts and alternations that could support insightful readings of the films as texts. Positions varied, however, on the extent to which such alternations needed to be anchored in the material distinctions drawn in a film. This methodological uncertainty echoes through most semiotic work of the period, supported in part by similar tendencies in earlier forms of analysis. Again, as Bordwell (1989: 80) for example sets out in some detail, traditional characterizations of a genre such as the Western as constructing alternations between good/evil, civilization/ wilderness and so on could just as well be found in the newly emerging semiotic descriptions. Whereas such work might consider itself ‘textual analysis’, for us any distance opened up between alternations that can be anchored in material distinctions – i.e., the physical ‘text’ – and alternations among interpretations (Hjelmslev’s connotative semiotics) needs to be considered

10

John A. Bateman & Janina Wildfeuer

more critically and, in fact, requires a far stronger methodological scaffold than was available at that time. Nevertheless, although immediately faced with challenges and boundaries, as we shall take up in a moment, the application of the notion of text analysis to film according to the foundation produced by Metz, and as illustrated in practice by Bellour and others by identifying paradigms expressed in a film’s technical features – particularly at that time in patterns of cutting and editing – was able to produce substantial results. Such analyses saw films in terms of their “process of production of meanings” (Heath 1973: 105) as affected by the ‘codes’ and structures of the filmic text that were posited. It is largely as a result of the precision of these analyses that they can often stand to this day – that is, regardless of any particular Freudian or other narratives against which, or through which, the films might have been read, the textual patterns identified remain and are, indeed, strongly suggestive of the need for interpretation. But the fledgling sets of methodological principles for guiding analyses articulated in these accounts also raised substantial foundational questions, and it was unclear where sources for potential answers might be found. Whereas the textual approach had taken much of its impetus from conceptions of ‘text’ within semiotics and linguistics, the frameworks available in those disciplines were still very limited – indeed, both disciplines were also only just beginning to emerge from the vortex depicted in Figure 1.1 and to engage more centrally with the phenomenon of text themselves. Basic theoretical constructs appealed to, such as ‘semiotic codes’, were inherently problematic as these were still generally conceived of within the Saussurean signifier-signified model along with its suggestion of a coding-decoding perspective on signs. This gives rise to some inherent contradictions. It is not possible, for example, to reconcile such a model with Metz’s account because of the gap that Metz deliberately opened up between physical manifestations, such as the particular content of shots, and the ‘larger structures’ that were to be placed over such manifestations by codes such as the grande syntagmatique. ‘Decoding’ in such a situation would be a strained metaphor at best, since the required ‘signifiers’ have no determinate form (and, even worse, no determinate signified). Metz consequently insulated the notion of codes from actual processes of coding-decoding on the part of film viewers and producers by anchoring them to the labor of the analyst, not the viewer. This position makes it difficult, however, to consider more empirically motivated investigations and, indeed, also falls back more on a ‘reading’, literarily-inflected interpretation model. These issues were taken up in particular detail in the context of film studies and its encounters with text analysis in Bordwell’s (1982) early critique of Metz’s use of ‘semiotic codes’. Bordwell pointed out that, first, Metz seeks only to talk of ‘sub-codes’ potentially relevant for film analysis without defining just what ‘codes’ those sub-codes may be subordinate

Introduction

11

to and, second, that Metz appears to accept ‘established wisdom’ as sufficient for their identification – we noted, for example, that discussions of film dating from Pudovkin and Eisenstein had already long talked of montage and shot scales; these were consequently accepted as relevant sub-codes. These criticisms are all apposite: their foundation is not, however, a particular weakness of Metz’s discussion but rather goes back once again to the lack of theoretical grasp at that time concerning how situated socio-historical development and apparently a-temporal ‘codes’ could be theorized at all. This situation held sway for a considerable period and is very much in evidence in subsequent broad rejections of the relevance of semiotic/textual accounts for film. As Bordwell expressed it in the mid1990s, for example: “Despite three decades of work in film semiotics […], those who claim that cinema is an ensemble of ‘codes’ or ‘discourse’ have not yet provided a defense of why we should consider the film medium […] as plausibly analogous to language.” (Bordwell 1996: 18) This was part of a broader critique of what was characterized as a dangerously anti-empirical and anti-historical ‘Grand Theory’ of film made up not only by structural linguistics, but also critical theory and psychoanalysis (cf. Bordwell 1996, 2004), all seen as contributing to ‘Theory’-driven symptomatic analyses of the kind problematized extensively in Bordwell (1989). In the terms we are developing here, we can characterize symptomatic analyses as making productive, some might then say too productive, use of the gap between material alternations manifest in film texts and levels of connotative interpretation. When there is no principled relationship between ‘codes’ (as langue) and the ‘texts’ manifested in use (parole), such gaps are pre-programmed to raise methodological and theoretical problems. It is now, therefore, time to return to the perspective offered by our graphic in Figure 1.1 and the various trajectories disciplines have taken since the broad reappraisal of the nature of text lying at its center. In particular, the disciplinary path followed within linguistics has addressed problems related to questions of ‘code’ at many points. Nowadays any notion of ‘code’, regardless of how it is defined, will be considered temporally anchored in sociocultural, historical practice. Bordwell’s earlier critique, therefore, sketches requirements that are now equally accepted and, indeed, constitutive of current linguistic theorizing about the nature of ‘texts’ and their interpretation.

2.2 … to Film as Text … This entire development played out within one of what have proved to be the most challenging areas of linguistic theory: that concerned with establishing the relation between Saussure’s langue and parole or between Hjelmslev’s

12

John A. Bateman & Janina Wildfeuer

(1961 [1943]) system and process/text. Whereas the intrinsic dynamics of textuality was seen in the 1960s and 1970s (and often still today) as a feature distinguishing more literarily-inflected accounts from the allegedly more static and ‘code-based’ models pursued within linguistics, more contemporary linguistic perspectives render arguments of this kind moot. The problems inherent to a separation of langue (static, synchronic) and parole (dynamic, usage-based) have been as clear within linguistics as in any other text-oriented discipline – arguably more so since the topic is so central to any understanding of the nature of ‘language’. As a consequence, it is now far more common in linguistic theory to embrace Hjelmslev’s insistence on the inseparability of system and process and to incorporate dynamics at the heart of the disciplinary construction of language and its use. As Michael Halliday describes it: “I prefer to think of these as a single complex phenomenon: the system only ‘exists’ as a potential for the process, and the process is the actualization of that potential. Since this is a language potential the ‘process’ takes the form of what we call text.” (Halliday in Martin 2013: 74) This is in fact necessary in any linguistic treatment of ‘text’ and has now received extensive theorization, even going back to reconfigure our readings of Saussure (cf. Thibault 1997). Any account or critique of film text analysis that is proposed today that makes reference to results in, for example, linguistics must be seen against this state of affairs and not the state of affairs back at the emergence of modern approaches to texts situated in the center of Figure 1.1. This then has important consequences for how we will choose to circumscribe ‘film text analysis’ for the current volume below. A different, but interestingly parallel, set of disciplinary trajectories unfolded within the more literary and culturally-inflected approaches depicted in our graphic. The vortex depicted in the center picks out the historical moment when ‘texts’ and notions of text entered their greatest state of upheaval. With the move intrinsic to the discussion of Barthes and others towards seeing texts as locations of cultural debate and divergent reading positions, the understanding and determination of filmic text changed also. Broader concepts of text were established drawing on and presupposing mutual relationships of interdependence between text, context, and culture (cf., e.g., Jahraus 2007). Accounts in film text analysis (e.g., particularly Ropars-Wuilleumier, Kuntzel: cf. discussion in Mowitt 1992; Rodowick 2001) naturally then came to voice similar concerns to those of Bordwell above, embarking on broad philosophical critiques of the relevance for film of Saussurean ‘code-based’ models. The style of argumentation employed in these critiques remained, however, in the largely metaphorical register opened up by Barthes and other literarily-inflected approaches. Developments of

Introduction

13

this kind were important and valuable, but also prone to methodological imprecision. Within this methodological trajectory, the material manifestations of film came to be engaged with only as far as was required to support more abstract, symptomatic interpretations, as it is only those interpretations that were seen as determining the pertinence, or not, of any specific filmic details taken into consideration for analysis. One consequence of this trajectory was that the analysis of film text-internal elements and meaningful units became less important; in their place, cultural systems and ideology, gender patterns, or philosophical questions as replicated in film became the driving forces of much of film theory and analysis. The filmic text could then be seen as a general source of evidence for social, historical, and cultural structures, but increasingly no longer underwent detailed textually or semiotically grounded analyses prior to, or accompanying, analysis. A less constraining understanding of ‘film as text’ thus spread across various disciplines and research areas, including not only affine areas such as cognitively oriented narratology, but also the broader contexts of media studies, cultural or postcolonial studies, philosophy, and psychology. It was then this form of ‘film text analysis’ that became predominantly associated with the term. Progressively distancing itself from the close reliance on material distinctions, interpretations came to be sought more in terms of psychoanalysis (Metz 1986) and, subsequently, social and cultural configurations (cf. Bordwell 1989: 73). Moves beyond the film text analysis of the 1960s and 1970s then again drew on critiques of earlier linguistic positions. Here, we meet precisely the same problematic of reoccurring calls within (not only) film text analysis to escape inappropriate constraints imposed by an assumed ‘logocentric’ semiotic model based on a ‘speech model’ ascribed to Saussure. Many of the discussions nearer the emergence from our vortex in Figure 1.1 (and unfortunately in analyses of those discussions in more recent work) reiterate these concerns and strive ‘instead’ to achieve dynamic models where the reading of a text is not ‘univocal’ but variable, manifold, and employing expressive resources beyond those provided ‘in language’ (cf. the theoretical discussions of film text analysis from Barthes, Ropars-Wuilleumier, Kuntzel, and others). In Ropars-Wuilleumier’s film text analysis, for example, re-orienting in the direction of Barthes’ re-appropriation of ‘writing’, we are told that: “The term ‘writing’, as always, requires comment: literally, it designates the graphic tracing of signs: linguistically, it is understood as the transcription of spoken language; but theoretically it introduces a critique of the model of signification involved in the linguistic and phonetic model of the sign.” (Ropars-Wuilleumier 1982: 147) More recent discussions also allude to this development, as in, for example, Rodowick’s (2001: 89) characterization of media change. Such concerns must

14

John A. Bateman & Janina Wildfeuer

be read very differently from today’s perspectives, however, since they have little relevance for more current models of signification, even within linguistics. Far more promising, we suggest below and throughout the contributions to this collection, is a realignment across the distinct trajectories suggested in Figure 1.1 so as to re-invigorate film text analysis with the more powerful methods and theories now available concerning text – particularly those pursued within the linguistics of text and discourse (cf., for discussion of this point, Wildfeuer in this volume). In the end, the style of discourse employed increasingly within film textual analysis of the later semiotic persuasion was not able to provide compelling accounts concerning the mechanisms of dynamic text interpretation and it fell to work emerging in other fields – such as cognitive film theory, for example – to focus on filmic narrative construction on the basis of inferential strategies. This work no longer particularly prioritized the role of the filmic text, however, and instead emphasized functions and stylistic conventions of filmic details working as cues for the recipient’s general cognitive capacity to comprehend film (cf., e.g., Bordwell 1985, 2011). It was then natural that the, in certain respects, broader understanding of ‘text’ that evolved in much film text analysis itself became a source of criticism for those committed to pursuing more detailed analyses of the film form. The increasing distance seen between the observable manifestations of film form and the explanations proffered invited far freer interpretations than could be motivated on empirical grounds – the situation particularly critiqued throughout Bordwell (1989). This state of affairs, together with the apparent claim that such analyses were building on some relation between ‘language’, semiotics, and film, also fed directly into Bordwell’s later rejection of the relevance of ‘codes’ and ‘discourse’ cited above. A textual analysis was then, somewhat ironically, rejected by exercising a mode of discourse that invited readings guilty precisely of not engaging sufficiently with the motivating factors of film and their organization, i.e., with their texts.

2.3 … to Dynamic Filmic Discourse We can see, therefore, that literary approaches to ‘text’, linguistic approaches to ‘text’, and even cognitive approaches to film have run up against some rather parallel concerns along their discipline-internal trajectories. The treatments of those concerns differ, but the recognition of the need to address the issues raised often constitutes a feature held in common. There is then certainly little to be gained by playing one off against the other – for example, by rejecting linguistic approaches because of their assumed adherence to a model of the sign sketched by Saussure but no longer found in any linguistic work, or by rejecting approaches to ‘text’ on the grounds that linguistics and semiotics require static codes in order to couch their model when ‘static codes’ are almost a historical footnote.

Introduction

15

In contrast, binding system and process together in the manner assumed in corresponding contemporary linguistic accounts makes treatments of text inherently temporal at several different scales, or ‘time depths’, all equally relevant and important for the treatment of film. The first unfolds within each individual text: texts develop in time, both in terms of their production and, even more significant, in terms of their reception – when participants engage in text ‘interpretation’, of whatever kind, this is a temporal, dynamic affair and can never be seen as static. The second scale relates to the system as a whole: the temporal unfolding here makes it clear that any talk of a system is always and irrevocably anchored in sociocultural time, it is never ahistorical. A system only ‘exists’ as the breaking wave arising out of the history of interactions and uses of the system up until that point. It is then precisely the developments that followed within the trajectory of linguistics and semiotics after their emergence from the ‘textual’ upheavals of the 1960s and the dividing of the ways across disciplines that need to be reactivated for film analysis. The contemporary situation for the trajectory of text within semiotics and, even more, within linguistics is now so different that it is important to be clear just how more recently developed views of text and discourse engage with the concerns of the early film text analyses, while also moving beyond much of the criticisms made both of those accounts and of those accounts’ critiques of others. In contrast to those earlier approaches, however, the current linguistically-inflected approaches to ‘text’ bring with them empirical methodological commitments to close analysis of material distinctions, of staying close to the text, as Dyer calls for. It is in this sense then that we will be re-working film text analysis and practice throughout all of the contributions of this book. Under this view, film text analysis involves a notion of text that is already radically multimodal (thus answering concerns of limitations to language by, for example, Ropars-Wuilleumier as cited in Mowitt 1992: 167), that is necessarily described in terms both of dynamic inferential process and at varying levels of abstraction and time-depths (thus addressing Bordwell’s concerns both of the need for inference and the historicity of codes), and that is grounded fundamentally in materiality and embodied perception (thus addressing concerns raised by Saussure’s lack of attention to non-arbitrary, iconic signs: cf. Wollen 1976). In summary, we suggest that new approaches in contemporary linguistics and multimodal analysis today make it possible to readopt and revise the examination of filmic characteristics as long as the locus of comparison between ‘film’ and ‘language’ is clearly placed at the level of the ‘text’ or ‘discourse’. We see this as one means of taking Dyer’s proposal further, while at the same time maintaining points of contact with cognitive and ‘literary’ approaches to film. In accounts of this kind, mechanisms for guiding and prompting the recipient, thus affecting and constraining his/her inferences during interpretation, are considered central (cf. Bateman/Schmidt 2012: 147; Bateman 2013; Tseng 2013; Wildfeuer 2014a: 14–17). Textuality is then of necessity concerned

16

John A. Bateman & Janina Wildfeuer

with ‘cues’ for interpretation; providing mechanisms for characterizing how such cues combine and guide interpretation is one of textuality’s main tasks. Crucially, text is no longer seen as a mere juxtaposition of (linguistic) signs in what is too often still assumed to be the ‘traditional’ semiotic sense of a product that has to be decoded, but rather as a constellation of dynamic interactions between recipients and the various textures in the artifact working as cues for guided and constrained interpretation. It is thus the complexity of internal relationships within a text, its texture and structure, that stands in the foreground of any description and which needs to be analyzed for its role in the meaning-making process; this theoretical orientation is taken up in its own right particularly by Wildfeuer in this volume. It is for this reason that we now suggest and re-emphasize text as a fruitful interface operating at various levels of description in film theory, a view which relies on a more focused notion of text and textual methodology than has become current. As we shall now see, this is taken up in each of the contributions to this collection. In the following, we briefly summarize each chapter with regard to their particular contributions to the book and the perspective taken to analyze an example film or filmic extract. This will suggest how a comprehensive and integrative approach to film text analysis makes it possible to examine film in a more explicit and at the same time reliable fashion.

3 Recent Perspectives of Film Text Analysis All the chapters in this book present recently developed methods for the fine-grained and systematic examination of how films mean. Despite their individual focus on particular aspects of filmic meaning construction, each approach follows the general idea of analyzing film as text by providing example analyses and demonstrations of how their frameworks can be applied to the concrete filmic artifact and help uncover textual structures and interpretative cues. The ordering of the chapters is structured to exhibit a gradual increase in the levels of abstraction considered, as well as in the breadth of the phenomena addressed, all seen through the perspective of film text analysis. Thus, while the first three chapters focus on less abstract and more fine-grained aspects of particular semiotic resources and their material manifestations (i.e., lighting as discussed by van Leeuwen/Boeriis; sound and space as analyzed by Huvenne; embodied action and dynamics as set out by Wildgen), the following two chapters move to discussions of larger patterns and some contrasting theoretical stances (i.e., a notion of cognitively-motivated narrative grammar set out by Cohn and an inferential, pragmatic perspective argued by Wildfeuer). The final two chapters then consider aspects of mediality and intermediality as such, again pointing to textual details as indicators of the interrelationships explored (i.e., a treatment of intermedial references as blending by Bateman; and various intertextual and intermedia viewpoints in Goggin’s discussion of

Introduction

17

the adaptation of a book into a film). Thus, by showing “transformative acts” of film analysis that, in the beginning, work from “‘bottom-up’ – mandatory, automatic psychological processes” on the basis of several resources on the material level and that, in the next step, bring in structures coming from “‘top-down’ – conceptual, strategic ones”, this arrangement exactly mirrors the interpretation process as outlined by Bordwell (1989: 3). In all cases, it is the “sensory data of the film at hand [that] furnish the materials out of which inferential processes of perception and cognition build meanings” (Bordwell 1989: 3). Our questions, and those raised individually and collectively by the chapters of the volume, are just how these components operate together and with which kinds of interrelationships. We now briefly point out in more detail the contributions of the chapters individually, before summarizing how these combine in the service of the kind of film text analysis we are motivating. Theo van Leeuwen and Morten Boeriis track this interpretation process in their discussion of lighting as a semiotic mode, suggesting a theoretical approach on the basis of the social semiotics of Halliday (1978) and Kress/ van Leeuwen (2006 [1996]). Following a detailed description of the history of film lighting as it moved from the experiments of pioneer cinematographers to more institutionalized and standardized practices and today’s lighting software, the authors set out a description framework for the various kinds of meaning realized by film lighting. Starting from the social semiotic principles developed in van Leeuwen (2005), they give a systematic overview of the rich meaning potential of lighting in terms of a ‘system network’ that classifies the potential at hand. This classification is itself organized with respect to the metafunctional diversification commonly adopted in social semiotic accounts, whereby meaning is characterized from the perspectives of what is represented, what relationships are enacted between participants in a communicative situation, and how the construction of these meanings is orchestrated in textual wholes. The authors thereby synthesize a range of approaches in multimodal film and media analysis that focus on descriptions of semiotic modes and their meaning potential. By exemplifying a variety of film lighting practices and illustrating each of these practices by a short example from a broad choice of excerpts from the whole history of film, van Leeuwen and Boeriis illustrate the applicability of linguistic and semiotic theories to the analysis of filmic technical details. Also focusing on very specific semiotic resources of film, Martine Huvenne addresses the concepts of sound and space. She investigates, in particular, the notions of spatial montage and the editing of spaces by analyzing sound as a key factor in the communication between film and recipient. By highlighting the important functions of sound and the auditory level of film, Huvenne goes back to and re-invigorates Eisenstein’s concept of film as an audio-visual composition, i.e. the explicit interaction of music, sound, and images as a unified form, while also highlighting the complex multimodal interplay of the various semiotic resources. This then aligns well with

18  John A. Bateman & Janina Wildfeuer recent work in multimodal film analysis (cf., e.g., Wildfeuer 2012, 2014b; Siefkes 2015). For her account, she provides a phenomenological approach to sound and listening on the basis of Merleau-Ponty (2002), placing a focus on the auditory space within audio-visual perception as a starting point for detailed analysis of several scenes from the film Gravity (Cuarón 2013). In the next chapter, Wolfgang Wildgen focuses on the reproduction of events and actions in filmic text as central points for an embodied understanding of a film’s (fictional or fantastic) story. By discussing the realistic physical grounding of these events and actions in the film’s material, i.e. its psychico-chemical production of the photo or photo frame, the author suggests the notion of movie physics as a foundational level of film analysis, going back to basic notions of the ‘mechanics of film’ by Hitchcock on the one hand and the concept of semiophysics (Thom 1990) on the other. Wildgen relates these concepts to the kinematics and dynamics of embodied action by analyzing example scenes from the James Bond movies Casino Royale, Quantum of Solace, and Skyfall. With this approach, he describes several levels of dynamism in filmic text, ranging from rigid frames and simple movements to the amalgamation of several movements and corresponding situations of catastrophe and chaos (used in their technical senses), thereby articulating some central patterns of filmic meaning construction based on the film’s material and technical details. Moving up the ladder of abstraction, Neil Cohn addresses the broad question of how people understand communicative sequential images, opening up the field of discussion to range across several types of media so as to add a comparative perspective between the dynamic artifact film and static visual narratives such as comics. By applying the concepts and framework of his Visual Narrative Grammar (VNG; Cohn 2013) to several scenes from the film Star Wars, Cohn argues that certain general principles of both form and grammatical narrative structures hold for both visual and audio-visual ­narratives. At the same time, differences in fundamental properties of the  media themselves (i.e., static vs. moving) create different ­affordances and therefore also lead to significant differences in their analysis. By these results and with regard to general cognitive ­mechanisms of ­narrative ­comprehension, Cohn concludes that it is useful to apply the Narrative ­Grammar framework ­further to cognitive accounts of film comprehension. Building further on the notion of narrative structure and a more ­discourse-oriented perspective on the filmic text, Janina Wildfeuer’s ­chapter then gives an overview of how multimodal, linguistic, and in particular discourse semantic approaches can be effectively brought together to gain more traction on the process of meaning construction. In particular, the author considers in detail how the recipient can construct meaning from the textual information given in the audio-visual material as well as on the basis of inferences drawn out of this material. Developments in more recent film semiotic and multimodal discourse analytical accounts are drawn upon

Introduction

19

to provide detailed definitions for the concepts of filmic text and filmic discourse, according to which the author distinguishes generally between semantic and pragmatic levels of description. She then illustrates with an example analysis of a scene from Gravity how these different levels of analysis support the description of the film’s materiality on the one hand, and the necessary interpretation steps in terms of abductive inferences on the other, in order to shed further light on film’s ability to guide and constrain recipients’ interpretations. John Bateman’s blending-based perspective on intermediality considers cases of media combinations in film. The chapter examines various contrasting film scenes relying on intermedial references in order to explore to what extent a formalized notion of blending offers a suitable analytical tool for revealing and motivating media references as aesthetic and design choices. By discussing example sequences all using blending as a mechanism for discourse integration, Bateman proposes a model of transferring and combining information and understandings across ‘domains’ cued by explicit textual indicators, including cases where the domains blended involve media-related information. This suggests a new contribution to the processes of inferential meaning-making during film interpretation. Example analyses of scenes from The Matrix, Shrek, Frost/Nixon, and others are discussed to show how the structural configurations and characteristics revealed by a formalization of blending can offer more discriminating descriptions of intertextual and intermedial relationships in filmic text. In the final analytical chapter, Joyce Goggin offers a case study of Elizabeth Gilbert’s novel ‘Eat Pray Love’ (2006) and its filmic adaption with the same title (Murphy 2010). Goggin analyzes this in terms of ‘expanded adaptations’, i.e. adaptations that go beyond the typical dyad of novel and film and that bring in further plot elements and texts from the surrounding context. By working out several thematic expansions of the book’s and film’s narrative in terms of current trends in self-help books and travel guides, as well as synthesized versions of Eastern religions, Goggin shows a basic intertextuality of both media texts at work. The interpretation of this intertextuality is highly dependent on several distinct knowledge sources based not only on the content of the original text and other book genres, such as travel guides, but also on various aspects of globalization and the appropriation of cultures and religions. With her discussion, Goggin highlights the importance of information that is in general not inherent to the filmic text, but has to be activated by certain narrative and intertextual structures.

4 Conclusion As has now been clarified, all chapters as summarized address issues currently of interest in international film studies on the one hand and the broad context of text studies on the other. With these case studies and example analyses, the book contributes to current discussions of film and text theory by

20

John A. Bateman & Janina Wildfeuer

providing insight into further new integrative approaches and tools for the analysis of filmic text. By explicitly focusing on the idea of film text analysis and its ability to provide a strong foundation for any examination of filmic understanding, we see the notion of film as text as a valuable interface for film theory in general. Such an interface is capable of bridging the gap between several disciplines and research areas all dealing with the question of how the detailed analysis of the meaning-making structures in filmic text can help us learn more about film, one of the most powerful audio-visual artifacts developed to date, as well as about its understanding by actual recipients. The book, therefore, brings together analyses at several levels of description, starting with basic semiotic patterns of individual filmic aspects, such as lighting or sound, and grammatical/cognitive as well as dynamic narrative structures. A more pragmatically oriented discussion of the film as a textual artifact then broadens the perspective towards discourse analytical questions such as blending and the film’s ability to adopt and reshape concepts and ideas from other media, as in adaptation. By combining these different approaches and their individual research foci under the label of film text analysis, the main idea of this collection is to combine descriptions of the technical devices on a lower level of analysis with broader interpretations of societal patterns of the film on higher levels. The detailed analyses in almost all chapters, as well as their theoretical points and accompanying frameworks, shed light on how this combination can offer a useful launching point for further work in this area.

Bibliography Barthes, R. (1977). “From Work to Text”. In: Heath, S. (ed.). Image – Music – Text. London: Fontana, pp. 155–164. Bateman, J. A. (2013). “Dynamische Diskurssemantik als allgemeines Modell der Semiose. Überlegungen am Beispiel des Films”. Zeitschrift für Semiotik 35(3–4), pp. 249–284. Bateman, J.A./Schmidt, K.-H. (2012). Multimodal Film Analysis. How Films Mean. London/New York: Routledge. Bellour, R. (1971). “Entretien sur la Semiologie du Cinéma”. Semiotica 4(1), pp. 1–30. Bellour, R. (1974). “The Obvious and the Code”. Screen 15(4), pp. 7–17. Bellour, R. (1975). “The Unattainable Text”. Screen 16(3), pp. 19–27. Bellour, R. (2000). The Analysis of Film. Edited by Constance Penley. Bloomington: Indiana University Press. Bordwell, D. (1982). “Textual Analysis, etc.”. Enclitic 6(1), pp. 125–136. Bordwell, D. (1985). Narration in the Fiction Film. London/New York: Routledge. Bordwell, D. (1989). Making Meaning. Inference and Rhetoric in the Interpretation of Cinema. Cambridge/London: Harvard University Press. Bordwell, D. (1996). “Contemporary Film Studies and the Vicissitudes of Grand Theory”. In: Bordwell, D./Carroll, N. (eds.). Post-Theory: Reconstructing Film Studies. Madison: University of Wisconsin Press, pp. 3–36.

Introduction

21

Bordwell, D. (2004). “Neo-Structuralist Narratology and the Functions of Filmic Storytelling”. In: Ryan, M.-L. (ed.). Narrative Across Media. The Languages of Storytelling. Lincoln: University of Nebraska Press, pp. 203–219. Bordwell, D. (2011). “Common Sense + Film Theory = Common-Sense Film Theory?”. David Bordwell’s Website on Cinema: Essays. May 2011. Online: http://www.davidbordwell.net/essays/commonsense.php [last accessed: 1 March 2016]. Bordwell, D./Thompson, K. (2001). Film Art: An Introduction. 9th Edition. New York: McGraw Hill. Cohen-Séat, G. (1948). “Le discourse filmique”. Revue Internationale de Filmologie 5, pp. 37–48. Cohn, N. (2013). The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images. London: Bloomsbury. Dyer, R. (2016). “The Persistence of Textual Analysis”. Talk given as part of the Kracauer Lectures in Film and Media Theory at Goethe-University Frankfurt, Germany, 26 January 2016. Video Online: http://www.kracauer-lectures.de/de/ winter-2015-2016/richard-dyer/ [last accessed: 01 March 2016]. Eco, U. (1976). “Articulations of the Cinematic Code”. In: Nichols, B. (ed.). Movies and Methods: An Anthology. Berkeley: University of California Press, pp. 592–607. Eisenstein, S. (1949). Film Form. Essays in Film Theory. Edited and translated by Jay Leyda. New York: Harvest Book. Gilbert, E. (2006). Eat, Pray, Love. One Woman’s Search for Everything Across Italy, India and Indonesia. New York: Riverhead Books. Halliday, M.A.K. (1978). Language as Social Semiotic: The Social Interpretation of Language and Meaning. London: Arnold. Heath, S. (1973). “Film/Cinetext/Text”. Screen 14, pp. 102–127. Heath, S. (1981). Questions of Cinema. Basingstoke: Palgrave Macmillan. Hjelmslev, L. (1961 [1943]). Prolegomena to a Theory of Language. Madison: University of Wisconsin Press. Jahraus, O. (2007). “Text, Context, Culture”. Journal of Literary Theory 1(1), pp. 19–44. Kress, G./van Leeuwen, T. (2006 [1996]): Reading Images. The Grammar of Visual Design. 2nd Edition. London: Routledge. Kuntzel, T. (1978). “The Film Work”. Enclitic 2(1), pp. 38–61. Martin, J. R. (ed.). (2013). Interviews with M.A.K. Halliday: Language Turned Back on Himself. London/New York: Bloomsbury. Merleau-Ponty, M. (2002). Phenomenology of Perception. London/New York: Routledge. Metz, C. (1966). “La Grande Syntagmatique du Film Narratif”. Communications 8, pp. 120–124. Metz, C. (1974a). Film Language: A Semiotics of the Cinema. Oxford: Oxford University Press. Metz, C. (1974b). Language and Cinema. The Hague: Mouton. Metz, C. (1986). The Imaginary Signifier: Psychoanalysis and the Cinema. Bloomington: Indiana University Press. Mowitt, J. (1992). Text: The Genealogy of an Antidisciplinary Object. Durham/ London: Duke University Press. Paech, J. (1988). Literatur und Film. Stuttgart: Metzler.

22  John A. Bateman & Janina Wildfeuer Pasolini, P.P. (1971). “Die Sprache des Films”. In: Knilli, F. (ed.). Semiotik des Films. Mit Analysen kommerzieller Pornos und revolutionärer Agitationsfilme. München: Carl Hanser Verlag, pp. 38–55. Pudovkin, V.I. (1926). Film Technique and Film Acting: The Cinema Writings of V. I. Pudovkin. New York: Bonanza Books. Rhodie, S. (1975). “Metz and Semiotics: Opening the Field”. Jump Cut 7, pp. 22–24. Rodowick, D. N. (2001). Reading the Figural, or, Philosophy after the New Media. Durham/London: Duke University Press. Ropars, M.-C. (1982). “The Graphic in Filmic Writing: A bout de souffle, or the Erratic Alphabet”. Enclitic 6(1), pp. 147–161. Ropars-Wuilleumier, M.-C. (1978). “Muriel as Text”. Film Reader 3, 262. Ropars-Wuilleumier, M.-C. (1981). Le texte divisé: Essai sur l’écriture filmique. Paris: Presses Universitaires de France. Siefkes, M. (2015). “How Semiotic Modes Work Together in Multimodal Texts: Defining and Representing Intermodal Relations”. 10plus1: Living Linguistics 1, pp. 113–131. Souriau, É. (1951). “La structure de l’univers filmique et le vocabulaire de la filmologie” Revue Internationale de Filmologie 7/8, 231–240. Thibault, P. J. (1997). Re-Reading Saussure: The Dynamics of Signs in Social Life. London: Routledge. Thom, R. (1990): Semio Physics. A Sketch. Redwood City: Addison-Wesley. Tseng, C. (2013). Cohesion in Film. Tracking Film Elements. Basingstoke: Palgrave Macmillan. van Leeuwen, T. (2005). Introducing Social Semiotics. London/New York: Routledge. Wees, W.C. (1973). “Dickens, Griffith and Eisenstein: Form and Image in Literature and Film”. The Humanities Association Review 24, pp. 266–276. Wildfeuer, J. (2012). “Intersemiosis in Film. Towards a New Organization of Semiotic Resources in Multimodal Filmic Text”. Multimodal Communication 1(3), pp. 233–304. Wildfeuer, J. (2014a). Film Discourse Interpretation. Towards a New Paradigm of Multimodal Film Analysis. London/New York: Routledge. Wildfeuer, J. (2014b). “Coherence in Film: Analysing the Logical Form of Multimodal Narrative Discourse”. In: Maiorani, A./Christie, C. (eds.). Multimodal Epistemologies: Towards an Integrated Framework. London/New York: Routledge, pp. 260–274. Wollen, P. (1976). “Cinema and Semiology: Some Points of Contact”. In: Nichols, B. (ed.). Movies and Methods: An Anthology. Berkeley: University of California Press, pp. 481–492.

Filmography Casino Royale (2006). Martin Campbell. Columbia Pictures. USA/UK. Eat, Pray, Love (2010). Ryan Murphy. Columbia Pictures Industries. Italy/India/ Indonesia/USA. Frost/Nixon (2008). Ron Howard. Universal Pictures/Imagine Entertainment/ Working Title Films/StudioCanal and others. USA/UK/France. Gravity (2013). Alfonso Cuarón. Warner Brothers. UK/USA. Matrix, The (1999). Lana Wachowski/Lilly Wachowski. Warner Bros. USA/ Australia.

Introduction  23 Quantum of Solace (2008). Marc Forster. Metro-Goldwyn-Mayer/Columbia ­Pictures. UK/USA. Shrek (2001). Andrew Adamson/Vicky Jenson. Dreamworks Animation. USA. Skyfall (2012). Sam Mendes. Metro-Goldwyn-Mayer/Sony Pictures. USA/UK. Star Wars (1977). George Lucas. Lucasfilm. USA.

2

Towards a Semiotics of Film Lighting Theo van Leeuwen & Morten Boeriis

1 Introduction In introducing Place and Peterson’s article on film noir, ‘Some Visual Motifs of Film Noir’ (1976), Nichols argues that it “fills a remarkable gap” in film theory by addressing how “the visual style shaped by camera movement, lenses, lighting and composition retains a fundamental and astonishingly poorly documented importance” (Nichols 1976: 325). The way in which Place and Peterson sought to fill that gap was by employing “the technical terminology commonly used by Hollywood directors and cameramen” as a step towards “the implementation of a (…) critical language” (Nichols 1976: 326). These terms, as summarized by Place and Peterson, cover four aspects of film lighting: (1) the function of lights in a typical ‘three-point lighting set up’ – a key light as the main source of illumination, a fill light to soften the shadows as required, so as to create either low key light (high contrast between light and shadow) or high key light (low contrast between light and shadow), and a back light, to separate figure from ground; (2) the distribution of light from each source, which can be concentrated in a narrow focused beam (spotted), or flooded in a broader diffuse beam; (3) the vertical angle of the light, which creates high direction and low direction, each with distinct shadow formations, and (4) highlight and shadow, as used in modeling surfaces. Since then, others have followed their approach, for instance, Bordwell/Thompson (1993: 152–157) and Monaco (2000: 197–202), who also focus on the ‘three-point lighting set up’ and the contrast between high key and low key. While such terms can be clearly defined, and therefore reliably used in analysis, they do not go very far in helping us understand how lighting makes meaning. When it comes to interpreting lighting, recourse is taken to the language of impressionistic adjectives. High key “gives an impression of reality in which the character’s face is attractively modeled, but without exaggerated or unnatural areas of darkness” (Place/Peterson 1976: 32). “Low key’, with its “rich, black shadows”, has “the potential to hide elements” and carry “connotations of the mysterious and the unknown” and “strange highlights” may create “sinister demented” qualities while low placed lighting may create a “stark lighting in which interior feelings of the characters are finally exposed and laid bare” (Place/Peterson 1976: 328–329). Bordwell and

Towards a Semiotics of Film Lighting

25

Thompson, similarly, describe the low key lighting in El Sur (Erice 1983) as “portraying an adult world full of mystery and danger”, the blue lighting in in Ivan the Terrible (Eisenstein 1947–1958) as “suggesting the character’s terror and uncertainty”, while the swordfight in Rashomon (Kurosawa 1950) “is intensified by the contrast between the ferocious combat and the cheerfully dappled lighting pouring into the glade” (Bordwell/Thompson 193: 157). We do not disagree with these interpretations. But from a semiotic point of view, many questions remain unanswered. What is the relation between particular formal and technical features of lighting and their meaning potential? To which degree are the meanings of lighting motivated by the nature of the lights, to what degree by the context in which they are used, for instance by the genre, or by the specifics of a specific story? In this chapter, we try to answer such questions by using a social semiotic approach, which, as van Leeuwen has argued in ‘Introducing Social Semiotics’ (2005: 3), seeks to inventorize semiotic resources and their history systematically. In particular, we investigate how lighting is used in specific contexts, including the way it is regulated, talked about, taught, critiqued, and so on, in those contexts. We will begin by sketching how film lighting developed into a semiotic mode, a resource for making meaning, and then present a systematic account, a ‘grammar’, of the semiotic resources which that history yielded and the meaning potentials they afford. In this way, we hope to take a further step towards documenting lighting and theorizing the ways in which it can be used to make meaning in film.

2 A Short History of Film Lighting In the early days of the cinema, lighting was not yet a semiotic mode, a meaning-making component of film production. The concern was to get enough light on the film, to get a decent exposure, especially because film stocks were not yet as light sensitive as they would later become. Hence, studios had a glass roof or even an open roof, and the light that illuminated the action was therefore even and ‘undramatic’. When there is no choice between different angles, different types of light and so on, there is no meaning. Meaning depends on difference.

2.1 Narrative Lighting Arguably, film lighting came into its own in the 1920s, when German Expressionist film-makers introduced, both literally and figuratively, ‘darkness’ into their films. In this, painters had preceded them by several centuries. Around 1500, they had begun to separate object brightness and illumination, adding highlights in white ink and shadows with black hatchings so as to “distinguish properties of the objects themselves from transitory effects momentarily imposed on them” (Arnheim 1974: 310). Leonardo is said to have been the first to make light “an active power from a given

26 Theo van Leeuwen & Morten Boeriis

Figure 2.1 Emil Jannings in The Last Laugh (Murnau 1924).

direction into a dark room […] sometimes fracturing the unity of bodies by tracing boundary lines across their surfaces” (Arnheim 1974: 32). It is exactly this that German Expressionist film-makers such as Murnau began to explore, and they did so for dramatic reasons. As Kracauer memorably puts it (2004 [1947]: 104), they sought to create “an unreal light that illuminates interior landscapes” and “discloses brightness and darkness within the soul itself, that eternal alternation of light and shadow characterizing the relations between human beings”. In a chapter on shadows, Lotte Eisner, similarly, argued that shadows, in German Expressionist cinema, represented “the dark side of the soul”, growing in size as that darkness takes over, as in Caligari, where the shadows of Cesare’s hands become larger as he becomes more murderous (Eisner 1955: 134). In Figure 2.1, Emil Jannings, as the once splendidly uniformed night watchman who has been demoted to toilet attendant, sits in a dark and glum room, with the bright light of the city outside, high up, out of reach, no longer part of his world. Classic Hollywood cameramen, too, studied lighting in the work of painters. As Lee Garmes, Director of Photography on films like Shanghai Express (von Sternberg 1932) and Gone with the Wind (Fleming 1940), said (quoted in Higham 1970: 35): “Ever since I began, Rembrandt was my favourite artist, […] and of course I have always followed Rembrandt in my fondness for low key.

Towards a Semiotics of Film Lighting

27

If you look at his paintings, you’ll see an awful lot of blacks. No strong highlights. You’ll see faces, and you’ll see hands and portions of clothing he specifically wants you to notice, but he’ll leave other details to your imagination.” They also valued first-hand observation and naturalistic motivation. James Wong Howe, Director of Photography on films by von Stroheim, Howard Hawks, Raoul Walsh and Samuel Fuller (quoted in Higham 1970) said: “Make all the sources of light absolutely naturalistic. If you are in a room and the scene is taking place at a certain time of day, try to find out where the light would come from, follow that, don’t impose an artificial style.” Thus, the art of film lighting, as it developed in Hollywood, was built on rigorous naturalism and observation as well as on departures from naturalism, whether for purposes of selective representation (‘things you want the audience to notice’) or for dramatic and symbolic effects. The same cameraman, (here Leon Shamroy, quoted in Higham 1970: 34) can at one moment emphatically state the importance of ‘accuracy’ (“if you walk into a room with a candle flame, only the area around the flame should have warm color, the rest of it should be cold”) and then, in the same breath, defend a departure from naturalism (“When you are shooting a sunset, use yellow light instead of white light and ignore realism, make a deliberate mistake”). In this period, lighting is not yet systematized. It develops incrementally, case by case. Cameramen “try to tell the story with light as the director tries to tell it with his action” (William Daniels, quoted in Higham 1970: 72), and constantly debate, experiment, and innovate. Nor were there rigid boundaries between lighting and other aspects of camerawork, such as choice of film stock and depth of field, or between lighting and other film-making disciplines, such as art direction. Arthur Miller for instance, Director of Photography for John Ford, Elia Kazan, and others, was known for his propensity to oil furniture to make it more reflective. When a new mode of expression is being forged, the pioneers of that new mode of expression “interrogate all the heterogeneous objects of which their treasury is composed to discover what each of them could signify and so contribute to the definition of a set which has yet to materialize”, as Lévi-Strauss (1966: 16–21) explained in his theory of bricolage. More recently, this approach to meaning-making has been theorized in relation to Gibson’s ecological theory of perception, in terms of sign-makers working with the affordances of available materials and processes to express what they need to express at a particular moment, and in a social semiotic approach to mode “equal emphasis is placed on the affordances of the material ‘stuff’ of the mode (sound, movement, light and tracings on surfaces, etc.) and on the work done in social life with that material” (Kress 2010: 80).

28 Theo van Leeuwen & Morten Boeriis

Figure 2.2 Still from Suspicion (Hitchcock 1941).

Stories from the classical period of Hollywood are full of examples of inventive sign-making which will later become part of the language of lighting, often involving non-technical accessories such as bits of black cardboard with holes cut in it, gauzes, veils, flicker sticks (sticks with bits of torn sheet hanging from them to move in front of an orange-filtered light so as to simulate the flickering of flames in an open fire) – and all this also in high budget films. In Hitchcock’s Suspicion (1941), Lina (Joan Fontaine) suspects Johnnie, her husband (Cary Grant), of poisoning her. As Johnnie brings her a glass of milk, Director of Photography Harry Stradling focused attention on that glass by putting a light in it, a contraption which had to be constructed specifically for that one shot (see Figure 2.2).

2.2 Textbook Lighting When film-making begins to be formally taught, and textbooks on lighting techniques begin to be published, more standardized, codified, generic concepts and methods of lighting begin to emerge, supposedly applicable to any story. The diagram in Figure 2.3, taken from a textbook written by cinematographer Freddie Young, describes the ‘three-point lighting set up’ in the language of general rules, and in terms of lighting’s contribution to ‘legibility’ rather than meaning: “The key light placed at an angle to the camera viewpoint strikes the subject obliquely and models its contours. The fill light adjusts the ratio of highlights and shadow contrast by raising shadow illumination. The backlight is an effect which may have many functions but usually enlivens the subject with small bright highlight areas and improves separation from the background.”

Towards a Semiotics of Film Lighting

29

Figure 2.3 Three-point lighting set up (Young/Petzold 1972: 107).

As Bourdieu (1977: 18) said, “In a pedagogical relationship, the master must bring to the state of explicitness, for the purposes of transmission, the unconscious schemes of his practice”. But, as he also noted, there may still be room for improvisation and adaptation to specific situations, and this in ways that usually remain implicit. The codification is only partial (Bourdieu 1977: 19): “The explanation agents may provide of their own practice, thanks to a quasi-theoretical reflection on their practice, which conceals, even from their own eyes, the true nature of their practical mastery as learned ignorance, a mode of practical knowledge not comprising knowledge of its own principles.” However, we believe that more generic forms of lighting did, in fact, begin to develop, especially in television drama. Figure 2.4a shows a scene from the crime series CSI Miami (2002–2012). The location is diffused and out of focus, and the lighting of the actors is difficult to interpret either from naturalistic motivation or from the exigencies of the story. What could cause or motivate that soft light on the faces of the two actors, but not on the hair of the woman front left? What could motivate that ‘small bright highlight

30 Theo van Leeuwen & Morten Boeriis

Figure 2.4 (a) Still from CSI Miami (left) and (b) screen grab from CSI computer game (right).

area’ on David Caruso’s head, so much stronger than the otherwise very bright background? Here lighting has become formal, merely concerned with modeling (‘legibility’) and ‘enlivening’, and perhaps also, as Elsaesser (2000: 44) suggested, with ‘eroticizing’ and ‘glamorizing’ objects and people by adding random touches of gloss and shine to the bare bones of the three-point lighting set up.

2.3 Computer-rendered Lighting Today, the results of such codifications are built into the lighting software that is now available for the production of film, architectural modeling, or games. Programs like MAYA or 3dsMax take codification a step further. Lighting is now a separate code, literally and figuratively, an explicit and circumscribed set of functions, a paradigm of choices, distinct from other aspects of photography and from art direction, costume design, makeup, and so on. If, in ‘textbook lighting’, the functions can still be modified or modulated by often ad hoc and improvised means related to the exigencies of specific scenes, this is now possible only insofar as the designers of the software envisioned these means. Here we do encounter something like a grammar of lighting, a system of automatized knowledge mediating between what we seek to express and how we can express it, but one developed by software designers. We can paraphrase Ellen Lupton, a writer on typography, who said “We do have a language of vision now, but it was developed by software designers” (quoted in Hunter 2006: 137). Yet, as Lev Manovich (2001) has forcefully argued, virtual images still strive for photorealism, and lighting is one of the key elements of the photorealistic rendering of computer-generated images, as it provides an illusion of the ever-changing, ever different nature of natural light, and hence, in this otherwise generic medium, of a specific time and place. And yet, despite this photorealism, many people experience a sense of unreality when looking at such images. Consider a screen grab from a CSI computer game (as given in Figure 2.4b). Looking more closely, it is not hard to spot those unrealities. What causes the wall underneath the larger window, and the figure on the left, to be so light, for instance? Why is there a shadow on the front of the piece of furniture the girl is leaning against and why is that piece of furniture not lit

Towards a Semiotics of Film Lighting

31

by whatever light illuminates her? Why does that light not spill on anything except her? And why does she not cast a shadow on the background? Looking at MAYA in a little bit more detail can help explain such feelings of unreality. We will give just one example, the types of light it provides. There are six basic types of light in MAYA: •

•

• • •

Point lights come from a non-existent and invisible point within the image and are omni-directional. Their effect, therefore, resembles that of the glass of milk in suspicion, except that it is unnecessary to hide the light source. It is itself invisible and can be put anywhere, even below the floor. No inventive ‘bricolage’ needed, just drag it to where you want it to be. Spotlights aim at a specific target and have a cone-shaped rather than an omni-directional beam. They can be spotted or flooded, and the beam can be made visible without any need for stirring up dust or smoke. Their effect, therefore, resembles that of non-virtual spotlights, but the ways in which they can be modulated – filtered, softened, etc. – is pre-determined. Directional lights have parallel rather than cone-shaped beams and are used as fill lights. Parallel beam lights do not exist in reality, but their effect resembles that of soft lights. Area lights project rectangular reflections, e.g. from a window, as given in the right of Figure 2.4. Volume lights pick out and light a particular volume, which makes the lit volume appear luminous rather than lit, as in the case of the girl in Figure 2.4b.

Furthermore, these lights do not create reflections or cast shadows: these have to be added separately to the elements affected by the light. This may be convenient, as it avoids the difficulty of unwanted reflections and shadows encountered in ‘non-virtual’ lighting, but it is also highly unrealistic. And as we have seen, light can come from impossible places, e.g. from below ground level. Even ‘negative lighting’ is possible: by typing in a negative value for the brightness of a virtual lamp, brightness is subtracted rather than added, which of course cannot be done in reality. The viewer may not explicitly realize all these things, but they nevertheless register. And yet the discourse of naturalism persists even here. Online MAYA tutorials recommend taking inspiration from reality, while at the same time embracing an aesthetic discourse in which the lighting has to ‘look cool’ – a variant, maybe, of Elsaesser’s ‘eroticization’ and ‘glamorization’.

2.4 Summary: Three Historical Stages of Lighting In sum, we have, so far, discerned three stages in the development of lighting: What we have called narrative lighting departs from the specifics of stories and their settings. It is naturalistically as well as symbolically motivated. It has fluid boundaries with other modes and media, art direction, costume

32 Theo van Leeuwen & Morten Boeriis design, makeup, and so on. It is learned on the job, through accumulated experience, and it contributes incrementally and gradually to the development of the resources that constitute the semiotic mode of lighting. What we have called textbook lighting presents general, or genre-specific, routinized principles (but will also acknowledge that ‘rules can be broken’ – a common rider in the teaching of creative practices). It is motivated by formal ‘image legibility’ criteria (‘separation’, ‘modeling’, etc.) as well as by aesthetic considerations, and it continues to adhere, at least in theory, to naturalism. It has stronger boundaries with other film-making disciplines. It relies on formal and semi-formal teaching, hence on knowledge invested in authoritative sources, and it allows the mode of lighting to be described in general terms, even though it still leaves freedom for improvisation and adaptation, and hence for gradual, organic development of the art of lighting. Computer-rendered lighting, finally, starts from a pre-designed set of possible types, modifications and combinations of lighting. It is motivated by formal, functional, and aesthetic definitions and considerations, and it continues to pay lip service to naturalistic lighting, even though it includes lighting options that are naturalistically impossible. It segregates different aspects of photography, and different aspects of rendering generally, and it is based on informal learning (workshops, tutorials, advice from friends, trial and error). This means that it is based on learning to ‘interact’ with externalized knowledge that is in fact authoritative as it cannot itself be interrogated, and codifies the resources of lighting in a way that can only be changed by software designers when a new version is produced. The system we will present in what follows is also systematic and explicit, but it is based on practice rather than precept, and aimed at providing tools for the analysis of lighting as a semiotic mode.

3 The Grammar of Lighting: Metafunctions Social semiotic descriptions of semiotic modes have traditionally taken communicative functions, rather than formal or technical features, as their point of departure. As in our earlier work (Kress/van Leeuwen 2006; Boeriis 2009), we will therefore take the three metafunctions, the three broad communicative functions posited by Halliday (1978), as our guide in this chapter, on the assumption that all semiotic modes will fulfill all three of these functions, but in their own ways, through the specific resources they have developed. This approach allows comparison between semiotic modes in terms of the semiotic work they do, while recognizing that different semiotic modes do so in different ways which themselves also carry meaning. It is an insight similar to that of Hollywood cameraman William Daniels when he said: “We tell the story with light, just as the director tries to tell it with his action” (quoted in Higham 1970: 72). It is also an important insight in an age where digital technology allows us, at any time, to choose whether to express what we want to express verbally or visually, and, if visually, whether through pictures or graphs and

Towards a Semiotics of Film Lighting

33

so on. In this chapter, we will, therefore, explore what lighting has in common with other modes as well as what is unique and specific about lighting. The three metafunctions are the ideational, the interpersonal, and the textual. The ideational metafunction is the ability of semiotic modes to realize representations of aspects of the world as they are experienced and understood in particular contexts. We, therefore, make the assumption that all semiotic modes can represent the world – though we do not assume that all semiotic modes can represent everything there is to represent in the world: what can be represented in a given cultural and historical context, and how it can be represented (e.g. concretely or abstractly, or both), is distributed across semiotic modes in a way that is itself culturally and historically specific. It is possible that in a world of ubiquitous electric power and luminous screens, light will become an increasingly important means of expression, in art, design, and architecture as much as in live entertainment and film. The interpersonal metafunction focuses on sign-makers’ views and feelings about the ideational content and the enactment of specific sign-making and sign-interpreting roles in interactions. Through lighting a film-maker can imbue a represented character or setting with a feeling of tender softness or harsh clarity, and through lighting a film-maker can blind or dazzle viewers, reveal or conceal things from viewers, and so on. The textual metafunction is the capacity of semiotic modes to form ‘texts’, complexes of signs which cohere both internally and externally with the context in and for which they are produced. As formulated in Kress/van Leeuwen (2006: 176), it comprises “the composition of the whole, the way in which representational and interactive elements are made to relate to each other, the way they are integrated into a meaningful whole.” Two further points should be made. While Halliday stressed that, in language, all three metafunctions operate at the same time, making a different though equal contribution to any instance of linguistic communication, van Leeuwen (1999: 189–191) argued that some semiotic modes are more biased towards the expression of ideational meaning, others to the expression of interpersonal meaning. In visual communication, for instance, interaction is carried by representation. To directly address a viewer with an image, image-makers must represent someone who looks at the viewer directly; they cannot look at the viewer themselves. In language, direct address is conveyed by the use of 1st and 2nd person pronouns such as ‘I’, ‘we’ and ‘you’. In the case of sound and music, on the other hand, representation may piggyback on interaction, as when two opera singers interact in a duet (in terms of who initiates, who responds, who interrupts, and so on) so as to create a representation of the relationship between two characters. But clearly such observations about the relative metafunctional strengths of different semiotic modes can only be made with metafunctional theory as a guide, in an exploration of whether, and to which degree, and how, a given mode can do ideational, interpersonal, and textual work. The metafunctional theory remains an indispensable heuristic.

34 Theo van Leeuwen & Morten Boeriis Secondly, even though in this chapter we depart from communicative functions, meaning is also made from the material properties of semiotic resources and the physical actions required to produce them. This is especially so when new semiotic resources are being developed, or existing ones renewed. In the case of lighting, for instance, a lamp can have characteristics that both literally determine what it can do (for instance, concentrate the light beam, so creating greater light intensity) and figuratively express what that can mean (e.g., the concept of ‘concentration’ or ‘intensity’, which can then be further ‘colored in’ by the context). Similarly the nature of actions that are part of articulating a lighting set up (for instance, placing objects in front of a light beam to cast distinct shadows) can give rise to meanings that are directly based on the nature of that action (for instance, ‘obscuring’, or ‘concealing’) – meanings which may then become more precise in a given context. Lighting operates both as a system in the result of the processes of codification we have described above and as a flexible and open set of resources because it is structured in terms of the functional elements that constitute any act of communication and in terms of the evaluative, value-laden meanings that arise from the way these functional elements are realized. We will, in this chapter, pay attention to both of these aspects of meaning-making.

3.1 The Grammar of Lighting: Formal Resources for Making Meaning with Light The formal features which function as resources for meaning-making with light are closely related to the technical light resources and do not as such carry distinct meaning in themselves but, rather, a meaning potential, which can be realized in the interplay of resources at play in the given context. As Place and Peterson show above, the basic ‘building blocks’ of lighting are in fact light and shadow and we operate with a basic distinction between three different ways in which lighting may appear in film: 1) as the play of light and shadow on a surface, 2) as a depicted or implicit light source such as the sun or a lamp, or 3) as beams of light in the air. This is further elaborated under the ideational metafunction below. Some core features constituting the semiotic resources of lighting (called ‘realization statements’ in social semiotics) are presented below. Brightness Especially bright objects come close to being sources of light themselves. What that brightness reveals can vary widely, but it will always radiate high intensity, high energy, high power, and strongly attract the viewer’s attention. Definition As mentioned, light can be hard or soft, and besides a mere indication of the size and character of the light source, ‘hard’ and ‘soft’, in turn, are rich

Towards a Semiotics of Film Lighting

35

sources of metaphor and therefore have a wide meaning potential, from the hard, impenetrable beauty of the ‘statuesque film noir heroine’ of Place and Peterson to the starkness of the contrast between the forces of light and dark in The Seventh Seal (Bergman 1957), and the soft, misty mystery of Dreyer’s Vampyr (1932).

Angle of Incidence The vertical lighting angle has been much commented on in the literature (cf.,  e.g., Millerson 1972: 121–123). When a light is placed low, pointing upwards to what it is illuminating, it creates shadows that project upwards instead of downwards. Given that the sun is above us, shadows naturally project downwards. The meaning potential of the ‘reverse modeling’ of low angle lighting is therefore mostly some sort of ‘abnormality’ – which, in certain contexts can acquire more specific meanings such as e.g. ‘bizarre’, ‘uncanny’, ‘horrific’ etc. The horizontal lighting angle has been less commented on in the literature but will be discussed in relation to the textual metafunction below. Modulation Film lighting causes modulation by projecting patterns which can either be random or diegetically motivated, suggesting bars, venetian blinds, etc., and which may be diffuse or sharply defined. Diegetically motivated patterns can invoke connotations, for instance of literal or figurative imprisonment or confinement (see Figure 2.2). More generally, modulation is always an act of selectively withholding visibility. Night interiors that are lit only by sources that enter the location from outside, such as the moon, or a street lantern, may be dark for the most part, with small random patches of light on walls or on shiny parts of the furniture. This can make familiar objects mysterious and cause disquiet and apprehension – anything can lurk in the shadows. (Random) modulation is also an important (quasi-)naturalistic device in the photorealistic rendering of computer-generated images, because of its ability to make flat surfaces less stark, less ‘clean’, less generic. Key Key is about the depth of the shadows, by contrast to the lit areas. In low key lighting, the contrast is high so that darkness with all its metaphoric potential can enter the thematics. In high key lighting, the contrast is low, making the light flat and even – the overall impression is then one of brightness, with all the metaphor potential that can bring. Color Colored light can be diegetically motivated – the warm light of a candle, the cold blue of a television screen lighting an otherwise unlit room – or used

36 Theo van Leeuwen & Morten Boeriis symbolically, often in scenes which portray subjective mental states such as dreams or intoxication. In Bergman’s Cries and Whispers (1972), the actors are bathed in red light, reflected from the interior walls of the manor house in which the film is set: “All our interiors are red, of various shades”, Bergman (1977: 60) wrote, “the whole thing is something internal. Ever since my childhood I have pictured the inside of the soul as a moist membrane in shades of red.” Movement Moving light, whether created by flames, torches or searchlights attracts attention. It is by nature unsteady and can, therefore, signify metaphoric unsteadiness – anxiety, foreboding, dizziness, and more. It also tends to light only parts of the scene which may create a sense of uncertainty and foreboding. Lighting is itself multimodal. Not only can light be more or less bright, it can, for instance, also be differently colored, or more or less defined – hard light throws sharply outlined shadows, and soft light throws softer, more diffuse shadows. Based on the principle of ‘experiential metaphor’ (van Leeuwen 2005: 29–36), such qualities of light can signal the ‘warmth’ or ‘coldness’, or the ‘softness’ or ‘hardness’ of characters, settings and actions in ways that can become more specific in specific contexts, as we will discuss in more detail below.

3.2 The Grammar of Lighting: The Textual Metafunction In our work on visual communication and film (Kress/van Leeuwen 2006: 175–215; Boeriis 2009: 203–241), we recognize three key dimensions of textual meaning: salience, framing and information value. All three can, we believe, also be realized by lighting. Salience is the ability of an image to attract the viewer’s attention to some elements in the image over others in their immediate environment, thereby signaling its semantic importance. This can be achieved by making such elements larger, more sharply defined, more strikingly colored and so on, but also, and importantly, by making them brighter (to be precise, by increasing the brightness contrast between such elements and their immediate environment), or by ‘pointing’ a visible light beam or reflection at an element in the picture (Boeriis 2009: 212–213). In Figure 2.5a, a still from Bergman’s Persona (1966), the lighting focuses attention on Elizabeth Vogler (Liv Ullman), an actress who has suddenly ceased to speak and is looked after by nurse Alma (Bibi Andersson). Alma, by contrast, talks incessantly, at first about everyday things, but then, as she begins to identify with Elizabeth, about intimate things, in the scene from which this still is taken about a sexual adventure she is, in hindsight, ashamed of. Throughout, it is sometimes Elizabeth who is in the dark while Alma is brightly lit, sometimes (as  in Figure 2.5a) the other way around. But other factors make Alma also salient, her being in the foreground, and on the right of the frame.

Towards a Semiotics of Film Lighting

37

Figure 2.5 (a) Still from Persona (Bergman 1966) and (b) Still from Kiss me Deadly (Aldrich 1955).

As film is an art of movement and visual changes over time will have a tendency to attract attention (Boeriis 2009: 214), a moving light will itself attract attention above any static light. Car headlights, for example, are used in the film noir classic Kiss me Deadly (Aldrich 1955), in which tough detective Mike Hammer gives a ride to a woman who has escaped from a mental institution (see Figure 2.5a) Different elements in an image may be salient to different degrees, creating a ‘reading path’ “beginning with the most salient element, and from there moving to the next most salient element, and so on” (Kress/van Leeuwen 2006: 204). As Zettl (1990: 39–52) also points out, salience hierarchies can “lead the eye” across the image and separate foreground and background. Framing divides a visual space into different areas, which, again, can be done in a number of different ways, for instance through frame lines formed by elements of the representation in an image or by some contrast between areas, including contrasts in lighting. An example of this can be seen in Figure 2.5a, where Liv Ullman is brightly lit while Bibi Andersson is sitting in a dark zone, framed by a vertical line formed by the curtain behind her. Such framings, as Boeriis/Holsanova (2012: 266) point out, group the elements within a given area as somehow belonging together, but also disconnect them to a greater or lesser degree from the other areas. In Figure 2.5a this enhances the figurative distance and disconnection between Elizabeth and Alma at this moment in the story. Like salience, framing occurs, one way or another, in all semiotic modes. In time-based media such as speech and music, salience is expressed by stressing sounds and framing by phrasing. This is also relevant to film, where framing is not only realized spatially, within images, but also temporally, by grouping shots into sequences – a break in the continuity of lighting can signal the beginning of a new sequence of shots. Information value is created by the positioning of elements in the visual space. It makes a difference whether an element is positioned in the upper or the lower part of the frame, on the left or the right, in the center or on the side (cf. Kress/van Leeuwen 2006: 179–200; Boeriis 2009: 217–225). Given the ‘landscape’ format of film, left and right placement is especially

38 Theo van Leeuwen & Morten Boeriis Information value

Salience

Polarized Not-polarized Minimum salience Maximum salience Temporal

Framing

Spatial Minimum disconnection Maximum disconnection

Figure 2.6 The textual meaning potential of lighting.

important, playing the same role as ‘before’ and ‘after’ in speech, where the beginning of a clause contains the theme that forms the point of departure of the message, and the end the New, that what is said about the theme, the core of the message. While the Given-New relation is realized by a polarized composition rather than by lighting, it can still play a role in interpreting lighting: to have a light Given and a dark New is not the same as to have a dark Given and a light New. In Figure 2.5a, Alma (Bibi Andersson) is the New, as well as being in the foreground and being in the dark – and this darkness is part of the meaning of the scene, in which she is confessing ‘dark’ moments in her past.1 In film, the narration often progresses from left to right in the frame, and light shining from the left then shines from ‘what has already been’ both in terms of the presented suyzhet (plot) and the fabula (story). Likewise, light shining from the right can be seen as coming from the ‘future’ of the story, from ‘what is to come’. Figure 2.6 summarizes this discussion in the form of a system network. Square brackets signify either/or choices, curly brackets ‘both … and’ options, and double headed arrows a graded (‘more or less’) rather than an absolute contrast.

3.3 The Grammar of Lighting: The Interpersonal Metafunction In our earlier work on visual communication (Kress/van Leeuwen 2006: 114–154; Boeriis 2009: 241–269), we considered three dimensions of interpersonal meaning, (1) interaction – what an image is doing to or for or with the viewer (we called that an ‘image act’, by analogy to ‘speech act’), (2) how an image positions the viewer in relation to what it depicts, and (3) the degree of naturalism of the represented (we call that ‘modality’ similar to the linguistic term). Such image acts, positionings, and naturalisms are understood from our experience of everyday non-verbal communication, whether firsthand or mediated. If, in everyday life, people perform speech acts, they will look at us in ways that suit these speech acts – sternly when they admonish, pleadingly when they beg, seductively when they flirt, and so on – and this may

Towards a Semiotics of Film Lighting

39

be accompanied by appropriate gestures, such as sternly raised index fingers, begging hands, pouting lips, etc. When such looks are represented in images, the gestures, and facial expressions, together with other elements of personal presentation, will all contribute to the expression of specific ‘image acts’. As for positioning, in everyday life we may interact with people at close distance, enacting a relation of intimacy (sometimes unwanted intimacy, as in an overcrowded buses or trains) or friendliness, or we may, literally and figuratively, remain at some distance from the people we interact with (cf. Hall 1966). In images, such relations can be suggested by size of frame, with close shots suggesting close relationships and long shots more distant and formal relationships. Again, in everyday interaction, we may face people or stand or sit beside them, without looking at them. In a lecture theater, students face the lecturer, even though their relation with this lecturer may be distant, but they do not face the other students, even though they are sitting close to them and may be their friends. In images, such relations are suggested by frontal versus profile angles – and various possibilities in between. Understanding such image acts and positions is therefore essentially asking: Who would I have to be, and where, in which kind of situation, to see this person (or this thing, or this landscape) in this particular way? A similar question can be asked in relation to lighting. Lighting, too, can be considered, not in relation to the lit object, but in relation to the viewer, realizing that lighting may either reveal people, things, places, and actions to the viewer, or conceal them from the viewer. Revealing is then, literally and figuratively, ‘throwing light on something’, which can of course be done to different degrees, either through uncovering the visible surfaces through partial illumination realized by the degree of the angle of the light or by clarifying structural details of depicted elements through adding lower or higher key as well as a different angle. Concealing can either take the form of obscuration, of not lighting an object, keeping it in the dark, or of bedazzling, pointing the light directly towards the viewers, so dazzling or blinding them. Obscuration will then acquire more specific meanings in specific contexts – it can frighten, mystify, exclude, and so on. And bedazzling can create disorientation, or fear of the unknown, as it does in everyday life. Much of this has also been noted by Place and Peterson. The main difference between their approach and ours is that we speak of a meaning potential that needs context for its complementation, rather than linking technical or formal features of lighting directly to specific meanings. In other words, we have a semiotic approach to the meaning-making, where we seek to describe the meaning potential grammatically. Naturalism plays a key role in the practices and discourses of lighting and has done so ever since lighting began to develop as a semiotic mode in German Expressionism. In social semiotics naturalism is discussed under the heading of ‘modality’, and seen as an interpersonal resource, resulting from social practices of validating representations as true or false, real or fake, and so on.2 In the theory of modality, the criteria for visual truth can be different in different contexts. Naturalism is defined as the ‘truth of perception’ (Kress/van Leeuwen

40 Theo van Leeuwen & Morten Boeriis 2006: 154–175; Boeriis 2009: 271–277) – the criterion for high naturalistic modality is the idea that what we see in an image looks more or less what we would see if we could see what is depicted in that image in reality, and this criterion plays a dominant role in mainstream lighting practices. Lighting is a natural for naturalism because it is transitory, momentary, ever-changing. In fact, judgments of naturalism are influenced by dominant technologies and practices. ‘Zero degree’ lighting, for instance, as described below, does not resemble what we would see in reality – in reality, it is not the case that everything is lit at a 45-degree angle from the side and from above. This kind of lighting is experienced as naturalistic. It has been ‘naturalized’ as the norm and as a neutral, self-effacing kind of lighting that just serves to illuminate things and make them clearly ‘legible’ without adding further meaning. Non-naturalistic light can also express a truth, a more abstract quality or a deeper meaning of what is being depicted. In earlier work we called that the abstract truth. Symbolic meaning potentials of this kind may result from open departures from naturalism, but lighting may also be at once naturalistically motivated and capable of signifying symbolic meaning, in what Panofsky (1953) has called ‘disguised symbolism’. Low angle light, for instance, may at once realistically represent the upward pointing shadows of a hand-held candle and convey the eerie un-naturalness typical of low angle lighting. The use of symbolic meaning potentials may express meanings relevant to specific moments in a film, or become characteristic of a genre or style of film-making, as in the case of German Expressionism and film noir. Finally, there is a form of lighting whose validity is based neither on naturalism nor symbolism, but on the aesthetic and sensory effect of lighting. In earlier work we called this ‘sensory modality’, and in film lighting it is realized by embellishing ‘zero degree’ lighting (see below) with touches of color, highlights, backlights, and so on, which give the image a richer, glossier, more glamorous look without adding much to the meaning. Zettl (1990: 32, 39–52) referred to this as the ‘emotional function’ of establishing or emphasizing mood or atmosphere, and Elsaesser (2000: 44) referred to it as a particular form of ‘neo-noirish’ television series lighting that ‘eroticizes’ and ‘glamorizes’ images. The network in Figure 2.7 summarizes the discussion in this section.

3.4 The Grammar of Lighting: The Ideational Metafunction The approach to the ideational metafunction of visual communication which we took in our earlier work (Kress/van Leeuwen 2006: 45–114; Boeriis 2009: 159–203), focused not so much on the meaning of the individual participants (that is, the individual people, places, and things depicted in images), as on the relations between these participants, in terms of the actions and events they are represented as being involved in, and in terms of the qualities and attributes they are represented as having, with actions and events realized by dynamic vectors (or, in film, by movement) and qualities and attributes realized by various compositional arrangements. In the case of lighting, the relevant relations are, on the

Towards a Semiotics of Film Lighting

41

Uncovering Clarifying Revelation

Minimum revelation Maximum revelation

Viewer relation

Obscuration Concealment

Bedazzling Minimum concealment Maximum concealment

Modality

Minimum modality Maximum modality Naturalistic modality

Coding orientation

Sensory modality Abstract modality Technological modality3

Figure 2.7 The interpersonal meaning potential of lighting3.

one hand, causation, the relation between light and its source, and on the other hand effectivity, the relation between light and the object or scene being lit. Like linguistic structure (Halliday 2004), lighting structure can be ergative, focusing on causality, or transitive, focusing on effectivity, and these two can of course be combined, as they are in the English language. Focusing first on causality, a light source may either be visible (agentive) or invisible (existential). ‘Existential’ light is presented as simply existing, simply being there, an indeterminate presence or atmosphere that pervades the entire scene. In the case of visible sources, light is agentive, an active agent doing the illumination. It includes natural light sources such as the sun and the moon, as well as artificial light sources such as candles, torches, festive lampions, neon lights outside a window, and so on, and both kinds of sources can introduce ideational meaning through the connotations they bring. Natural light sources can, in addition, denote different times of the day or weather conditions through the length and definition of shadows and the color temperature of the light. Again, in night interiors we may see the light that illuminates an actor, as in Figure 2.5a, or infer it through the shape of the circle of light cast on the wall or on a desk, and night may also be suggested by a gradual tapering off of the intensity of the light towards the top of a wall in the background, as if caused by a shaded lampstand, but without this lampstand being shown. And, as we will discuss below, both visible and invisible light sources may have qualities that lend themselves to symbolic interpretations.

42 Theo van Leeuwen & Morten Boeriis

Figure 2.8 (a) Still from The Seventh Seal (Bergman 1957) and (b) Still from The Magician (Bergman 1958).

Light may also emanate from luminous objects, and these may be everyday luminous objects such as television screens or computer monitors, or objects we do not normally regard as light sources, so that their luminosity will be interpreted as ‘not natural’ or ‘not normal’, which may then, in context, acquire specific meanings. As a source of light the glass of milk in Figure 2.2, for instance, is ‘not normal’, suggesting the milk is poisonous. The chess game in Bergman’s The Seventh Seal (1957) is ‘not natural’ because it has the power to decide over life and death – in the film, Max von Sydow is a knight who plays a game of chess with a personification of Death (see Figure 2.8a), whose face could also be seen as luminous, as it is not shaded in the same way as Max von Sydow’s face. Finally, the light itself can be visible, as a light beam, or a diffuse atmosphere or fog. We will refer to this as the materialization of light. Figure 2.8b is from Bergman’s The Magician (1958), with Max von Sydow as a magician with supernatural powers. When a light source is not visible, it may still be identifiable, either by inference, on the basis of length and depth of shadows or the shape of reflections, or by reference to some other shot which will show, for instance, the window through which light is entering or the desk lamp that illuminates an actor, and which may either have featured in an earlier shot or be revealed in a shot that is still to follow, in what Zettl (1990: 32) has called ‘predictive lighting’. Linguists refer to these forms of identification as ‘anaphora’ when the identity of something is retrieved from the preceding text and as ‘cataphora’ when it is retrieved from the following text. Tseng (2013) has used these and other related concepts in her theory of filmic cohesion. However, as we discussed in relation to Figure 2.4, it is not always possible to even infer what the source of a light might be, in which case a diegetic light source no longer motivates the lighting.Textbook lighting has introduced a form of lighting that cannot readily be explained naturalistically, yet does not strike audiences as unnaturalistic in the way Figure 2.8a does, where there appear to be two moons. Zettl speaks of an ‘organic’ function of lighting and considers it realistic (1990: 39–52), but it may be better to consider it a neutral, ‘zero degree of lighting’ which functions as the norm and therefore

Towards a Semiotics of Film Lighting

43

Agentive Illumination Causality

Luminosity

Existential Natural Artificial

Light materialization

Effectivity

Diegetic lighting Non-diegetic lighting

Zero-degree lighting Symbolic lighting

Figure 2.9 The ideational meaning potential of lighting.

as ‘normal’, despite the lack of diegetic motivation. By reference to this ‘zero degree’, we can identify departures from the norm that are still not naturalistically motivated but, rather, symbolically motivated. Figure 2.9 summarizes the ideational meaning potential of lighting as discussed in this section.

4 Coda We have, in this chapter, attempted a principled approach to film lighting, which, we hope, has demonstrated that a metafunctional approach can help elucidate how lighting makes meaning in films. Nevertheless, it is also clear that specific qualities of lights and lighting are not uniquely related to specific metafunctions and can realize ideational, interpersonal, as well as textual meanings. Brightness, for instance, plays a key role in realizing salience and framing, that is, in realizing textual meanings, but it is equally important in realizing ideational meanings, and the ideational meanings it can realize may become interpersonal when linked to direct address, to the look at the camera. Perhaps this shows that, despite its codification in systems like MAYA, lighting is still a developing craft whose functions have not as yet yielded separate means of expression or perhaps it shows that resources today interact in very complex ways in multimodal texts where modes are not being as clearly delimited as they once were: “We seem to be at an odd moment in history when frames are dissolving everywhere, and formerly clear boundaries are becoming ever more blurred. It is not, therefore, surprising that the same may be happening with representational resources.” (Kress/van Leeuwen 2001: 125) It is becoming clear, however, that lighting is an increasingly important semiotic mode in today’s world, and we hope that our exploration of film lighting will also be of use in the study of lighting in other domains – the abstract arts, theatrical shows of all kinds, ceremonies and festivities, interior decoration, architecture, and the design of everyday objects, including toys. Film

44 Theo van Leeuwen & Morten Boeriis lighting, and indeed all the other semiotic modes that come together in this multimodal art form, should not be treated in isolation from the overall semiotic landscape of which film is part, and in relation to which its semiotics will ultimately be most fruitfully studied.

Notes 1. ‘Reading Images’ (Kress/van Leeuwen 2006: 194–6) also discusses Center-Margin and Ideal-Real (top-bottom) relations. Although many examples can be found in which lighting focuses on a central element in the composition, usually there are no distinct ‘marginal’ elements, and the lit central element is set off against a dark background. Vertical directionality has already been discussed in the section on ‘angle of incidence’ above. Continuity should also be regarded as a textual resource in film, and lighting does play an important role in continuity. Maintaining the logic of a lighting set up throughout a scene that is unified in time and space provides continuity, and any break in that continuity must either be motivated, e.g., by a character in the film switching on a light, or interpreted as subjective, supernatural etc. 2. The social semiotic approach to modality derives from the work of Halliday (1994) where the link between modality and interpersonal meaning is argued on grammatical grounds, since modality and polarity are closely interwoven with the mood structure of clauses, rather than on sociological grounds, as, for instance, in the work of Hodge/Kress (1988). 3. In social semiotics, the term ‘coding orientation’ is used to refer to modality criteria that dominate in specific contexts. Apart from naturalistic, abstract (symbolic) and sensory modality criteria, social semiotics also recognizes a ‘technological’ coding orientation in which the criterion is pragmatic, and the image of higher modality to the degree that it can play a useful role in specific practice, e.g., research projects or technological production. Arguably lighting can also be technologically motivated, for instance in macro films of product details, films from microscopes, medical documentation films, endoscopic films, research documentation, and so on.

Bibliography Arnheim, R. (1974). Art and Visual Perception. Berkeley/Los Angeles: University of California Press. Bergman, I. (1977). Four Stories by Ingmar Bergman. New York: Doubleday & Co. Boeriis, M. (2009). Multimodal Socialsemiotik & Levende Billeder. Odense: Syddansk Universitet. Det Humaniske Fakultet. Boeriis, M./Holsanova, J. (2012). “Tracking Visual Segmentation: Connecting Semiotic and Cognitive Perspectives”. Visual Communication 11(3): pp. 259–281. Bordwell, D./Thompson, K. (1993). Film Art – An Introduction. 4th Edition. New York: McGraw-Hill. Bourdieu, P. (1977). Outline of a Theory of Practice. Cambridge: Cambridge University Press. Eisner, L. (1955). Die dämonische Leinwand. Frankfurt: Fischer Verlag. Elsaesser, T. (2000). Weimar Cinema and After: Germany’s Historical Imaginary. London: Routledge.

Towards a Semiotics of Film Lighting  45 Hall, E. (1966). The Hidden Dimension. New York: Doubleday. Halliday, M.A.K. (1978). Language as Social Semiotic. London: Arnold. Halliday, M.A.K. (1994). An Introduction to Functional Grammar. 2nd Edition. London: Arnold. Higham, C. (1970). Hollywood Cameramen: Sources of Light. London: Thames and Hudson. Hodge, R./Kress, G. (1988). Social Semiotics. Cambridge: Polity. Hunter, L. (2006). “Critical Form as Everyday Practice – An Interview with Ellen Lupton”. Information Design Journal 14(2): pp. 130–137. Kracauer, S. (2004 [1947]). From Caligari to Hitler: A Psychological History of the German Film. Princeton: Princeton University Press. Kress, G. (2010). Multimodality: A Social Semiotic Approach to Contemporary Communication. New York: Routledge. Kress, G./van Leeuwen, T. (2006). Reading Images – The Grammar of Visual Design. Second Edition. London: Routledge. Lévi-Strauss, C. (1966). The Savage Mind. Chicago: University of Chicago Press. Manovich, L. (2001). The Language of New Media. Cambridge: MIT Press. Millerson, G. (1972). The Technique of Lighting for Film and Television. London: Focal Press. Monaco, J. (2000). How to Read a Film – Movies, Media, Multimedia. 3rd Edition. New York: Oxford University Press. Nichols, B. (1976). Movies and Methods. Vol. I. Berkeley/Los Angeles: University of California Press. Panofsky, E. (1953). Early Netherlandish Painting. New York: Harper and Row. Place, J. A./Peterson, L.S. (1976). “Some Visual Motifs of Film Noir”. In: Nichols, B. (ed.). Movies and Methods. Vol. I. Berkeley/Los Angeles: University of California Press, pp. 65–75. Tseng, C. (2013). Cohesion in Filmic Text: Tracking Film Elements. London: Palgrave Macmillan. van Leeuwen, T. (1999). Speech, Music, Sound. London: Palgrave Macmillan. van Leeuwen, T. (2005). Introducing Social Semiotics. London: Routledge. Young, F./Petzold, P. (1972). The Work of the Motion Picture Cameraman. London: Focal Press. Zettl, H. (1990). Sight Sound Motion – Applied Media Aesthetics. Belmont: Wadsworth Publishing Company.

Filmography Cries and Whispers (1972). Ingmar Bergman. Cinematograph AB. Sweden. El Sur (1983). Victor Erice. Chloe Productions. Spain/France. Gone with the Wind (1940). Ian Fleming. Selznick International Pictures. USA. Ivan the Terrible (1947–1958). Sergei M. Eisenstein. Mosfilm. Soviet Union. Kiss me Deadly (1955). Robert Aldrich. Parkplane Pictures Inc. USA. Magician, The (1958). Ingmar Bergman. Svensk Filmindustri. Sweden. Persona (1966). Ingmar Bergman. Svensk Filmindustri. Sweden. Rashomon (1950). Akira Kurosawa. Dalei Motion Picture Company. Japan. Seventh Seal, The (1957). Ingmar Bergman. Svensk Filmindustri. Sweden. Shanghai Express (1932). Josef von Sternberg. Paramount Pictures. USA. Vampyr (1932). Carl Theodor Dreyer. Tobis Filmkunst. Germany/France.

3

Editing Space as an Audio-Visual Composition Martine Huvenne

1 Introduction In this chapter, I investigate the notion of filmic space, spatial montage, and the editing of spaces in film with an analysis of Gravity (2013) by Alfonso Cuarón as a case study focused on the way sound evokes and creates the ‘lived space’ of Dr. Ryan Stone. As a public, we ‘resonate’ with her experiences, situated at the center of filmic space through sound. Even though the visuals are amazing in Gravity, I argue that sound and the sound concept are key factors in the communication with the public. The sound is mixed in an elaborate dynamic space in order to give the audience an immersed experience. This brings us back to Eisenstein’s concept of film as an audio-visual composition: an interaction of music, sound, and images as a unified form. Already in 1940, Eisenstein stated his intention that the music of ‘The Ride of the Valkyries’ (Richard Wagner 1856) should surround and envelop the audience by means of multiple speakers. These loudspeakers would be phased so that the sound would give the impression of riding backward and forward across and around the auditorium and its surrounding spaces (Robertson 2011: 69–70). Unfortunately, I haven’t had the opportunity to listen to Gravity in a Dolby Atmos theater. Nevertheless, the concept of the auditory space is fascinating, and I propose a phenomenological approach to sound and listening with a focus on the auditory space within audio-visual perception as a starting point for the analysis. This includes the questioning of four topics: filmic space, the way sound evokes and creates space in audio-visual perception, listening, and film as an audio-visual composition.

2 The Sound Concept of Gravity Gravity tells the story of Dr. Ryan Stone (played by Sandra Bullock) on her first shuttle mission, with veteran astronaut Matt Kowalski (played by George Clooney). A seemingly routine mission turns into a disaster when debris flies along destroying the shuttle. Stone and Kowalski are left completely alone – tethered to nothing but each other and spiraling out into the blackness. They have lost any link to Earth – and any chance of rescue. Fear

Editing Space as an Audio-Visual Composition

47

turns into panic when almost no oxygen is left. The only way to get home is to go further out into the terrifying expanse of space.1 As an audience, we mostly listen with the ears of Stone. But at the same time, we hear her breathing, we hear the (non-diegetic) music, and we also connect our listening to the visual space on screen. We are ‘in’ and ‘out’ of her situation. Through the sound, we enter into her experiences of the claustrophobic environment in the cabin, her dialogues with her co-astronaut Matt Kowalski, her floating in space, her mediated connectedness with the earth. Sometimes we perceive her also from a distance, accompanied by the film music. The fact that the sounds are mostly conceived from Stone’s bodily perspective gives a feeling of an anthropocentric orientation in which “the human body is the measure of direction, location, and distance” (Sobchack 2004: 14). The sound does not give any insight into a Euclidian organized and directed space but informs us about the relation between the main character and the objects and humans surrounding her. From the start of the creation process, it was clear that the sound design in this film should be different. With Glenn Freemantle as Sound Designer and Supervising Sound Editor of the film, Nicolas Becker as Foley Artist, and Skip Lievsay as the sound re-recording mixer, the director Alfonso Cuarón opted for an immersive sound experience of the film. The music composed by Steven Price is mixed in such a way that the audience is moved and disorientated through the spatialization of the music: i.e., the way the music is distributed in space plays a crucial role.2 In an interview with Michael Coleman, Cuarón and sound mixer Skip Lievsay explain that their sound concept started from the main character, Dr. Ryan Stone: “Sound is constantly traveling, it is very dynamic. It is geographically very literal: if a character is behind you, the sound is coming from behind.”3 Sound supervisor Glenn Freemantle adds: “the concept came at that point that there is no sound in space: so we decided to approach the sound through concepts of touch and listening through vibrations.”4 Touch, vibrations, heartbeat, radio signals, and breathing are all part of making a connection with the main character. The relation between the main character and her environment is taken into account at any given time. Each time she turns, the sound turns with her. And we as the audience also turn with her through the sound. In other words, the auditory space doesn’t start from what is seen on screen, but from the centralized bodily position of the main character in a space and this without clear (visual) spatial references. In Gravity, every sound is a record of the particular ‘hearing’ of the main character. Sometimes sounds are in front of her, or sometimes behind her, at her back.5 Sometimes we listen with her to the radio voices or to her co-astronaut Matt Kowalski. Some of the sounds are accentuated and worked out in detail, starting from her situation, her perception, and her feelings. This evokes and creates a dynamic space and a dynamic, active audience,

48

Martine Huvenne

moving with the sound. Each time the main character is turning around the sound turns with her. And with the sound, the audience turns around. The sound concept of Gravity is innovative for a mainstream film: instead of thinking sound in relation to the image, the sound concept started from the ‘here’, the situatedness6 of the main character who is floating in a very dynamic space without visual references. This had consequences not only for the sound recording but also in the mixing. The sound mixer of Gravity, Skip Lievsay, explains: “The other side of the dialogue question was the panning. Alfonso (Cuarón) likes the idea that the words are attached to people. […] It removes a certain veil of film shenanigans where the old simple idea is that all dialogue comes from the center channel – which is a film construct. It doesn’t happen that way in life, and it isn’t really necessary to do that in the modern film formats. […] Even though the actors on camera turn and look to that direction, and then the camera spins over to see the person entering, most filmmakers will argue with you that that is a distraction and that you need to hear it on the center channel, or maybe on the left side – maybe. And I think one of the great things about cinema is the presumption that things are happening outside of the proscenium. That the film is pointed in one direction, but everything else is still going on outside of that frame.”7 Surroundabilty is thus extremely elaborated in the sound design of Gravity. A special sound mix was created for Dolby Atmos, taking full advantage of the amazing nuance, spread and separation of the speakers on the ceiling, and the total range of all the speakers in the room. The purpose was to create a sound experience in which the audience is fully immersed in an aural journey.8 Sound is guiding this audience through a dynamic and multi-layered space. The visibly immobile audience sitting in a seat is always moved, is always mobile and traveling through the sound. In the following, rather than focusing on the technical and innovative aspects of this sound design, I would like to focus on the perception of the filmic space in this film.

3 Filmic Space Anchored in a Lived Space Starting the sound concept from the auditory perception of the main character, with the knowledge that only the sounds physically felt by this character are audible in the film, questions the role of sound in the creation of the filmic space. With the auditory perception of the main character as the basis of the sound design and the fact that there is no sound in space, so what we see doesn’t make any sound, the auditory field has to be investigated. It is important to define which kind of space we are speaking about and how this space is created or evoked.

Editing Space as an Audio-Visual Composition

49

In a lecture in Brussels (in March 2007), the French sound engineer and sound artist Daniel Deshays explained: “Sound is not conscious. It is the lived (le vécu). In cinema, the sonorous (le sonore) is the body of the film. It is the flesh (the substance: la chair) of the film. It works on the intensities, the velocity, the flux. The sonorous works on our sensibility, on the affects. The sonorous inscribes itself in the body of the audience. There is no global listening: we are traveling in the different sounds.” This means that we also have to mention that we are moving through this auditory anthropocentric spherical space in a dynamic embodied way. Being at the center of this space, we do not have an overview. Space is no longer described and defined from an external standpoint, an overview, but as something that is constituted through experience. Does not the experience of space provide a basis for its unity by means of an entirely different kind of synthesis (cf. Merleau-Ponty 1957: 284)? – The phenomenologist Maurice Merleau-Ponty has posed this question and I take it as the basis of my proposition to unify the different auditory spaces in one space experience for the audience. It is the audience who brings the different experienced auditory spaces together. But first, I go back to the question: which kind of space are we speaking about and how is this space created or constituted? In her book “The Skin of the Film” (2000), media theorist Laura Marks makes the distinction between optical and haptic space following the ideas of Deleuze and Guattari about striated and smooth spaces, geometrical and nomadic spaces. “I arrived at it (haptic space) from Gilles Deleuze and Félix Guattari’s concept of “smooth space”, a space that must be moved through by constant reference to the immediate environment, as when navigating an expanse of snow or sand. Close-range space is navigated not through reference to the abstractions of maps or compasses, but by haptic perception, which attends to their particularity. Deleuze and Guattari’s privileged agents of haptic perception are nomadic people, such as Bedouins and Inuit. They write, “It seems to us that the Smooth is both the object of a close vision par excellence and the element of a haptic space (which may be as much visual or auditory as tactile). The Striated, on the contrary, relates to a more distant vision, and a more optical space – although the eye, in turn, is not the only organ to have this capacity.” (Marks 2009)9 Interesting to know is that the terminology of smooth space and striated space at first were terms to indicate different perceptions of contemporary music by the French composer Pierre Boulez (°1925). In his book “On Music Today” (1971) he introduces the term smooth space (espace lisse) to indicate

50

Martine Huvenne

the perception of contemporary music without having any references of a musical space. For example, in an atonal or serial musical composition, is it possible that the listener is ‘lost’. He or she misses the overview of the musical space and is thus, obliged to follow each moment that is presented in the music in a ‘nomadic’ way. This ‘being lost’ is also described by film theorist and phenomenologist Vivian Sobchack, who compares the topological lived space with the space of a child’s world in which the survey of the subject/horizon/world relationship contrasts with the model of the Renaissance perspective. Sobchack describes the lived space as a field or an environment, a ‘milieu’ in which the body is positioned in the center of a surrounding world; thus the horizon is not flat but radially curved (cf. Sobchack 2004: 20). This brings us also to Don Ihde’s approach to sound and listening. For him the auditory field is bi-dimensional: being both spherical and directional. Being at the center of the surrounding world is a characteristic of the auditory field different from the position of the viewer who looks at something: “Were it to be modelled spatially, the auditory field would have to be conceived of as a “sphere” within which I am positioned. […] The field-shapes of sound include both directionality and surroundability.” (Ihde 2007: 75–77) In this phenomenological approach, sound and listening are correlated. There is no sound without listening, no listening without sound. The perception of a sound depends thus on the listening strategy: the way we direct (or just not direct) our attention in listening. Applied to Gravity, it seems that the auditory space has taken the upper hand to transmit the experience of the film. The dominance of the visual optical film space diminishes as a function of the lived space of the audience (and of the characters) evoked by sound and images. Starting from the perspective of the main character in the sound design, we experience exactly what Ihde describes: sometimes we listen with a directed attention to sound and sound seemed placed in relation to what we see, but other times, we are totally immersed through the sound, also because of the importance of surroundability as described in the sound concept.

4 Thinking the Interior In “Film Art: An Introduction” (2010), film theorists David Bordwell and Kristin Thompson present analytical tools to question film sound. In their conclusion on sound and space, one of the questions they put forward is about diegetic or non-diegetic space: “Where is the sound coming from? In the story’s space or outside?” (Bordwell/Thompson 2010: 306). They state that the spatial dimension of sound is defined through the source of the

Editing Space as an Audio-Visual Composition

51

sound. “Our beliefs about that source have a powerful effect on how we understand the sound” (Bordwell/Thompson 2010: 284). And they make a distinction between ‘diegetic’ having a source in the story world and ‘non-diegetic’ sound represented as coming from a source outside the story world. Applied to Gravity, we can say that every sound is related to the story world with the main character at the center of the story. Only the film music can be indicated as non-diegetic. However, being in the center of ‘a space without sound’, the representation of the sound and the spatial dimension defined through the source can be questioned: “From a hearing-centered standpoint, sound is inherently spatial because the process of audition attaches a spatial ‘narrative’ to each sound (Altman 1992: 19); from a vibration-centered standpoint, sound does not exist without its propagation in space (Henriques 2010).” (Eisenberg 2015: 193) With these two quotes, Eisenberg considers sound from its ‘visual source’. As has already been said: the lived space of the main character Stone comes into account. And a lived space is created through all senses. Ryan’s (visual) movements are represented audibly in the film not because we see the source, but because she can be aware of them, hearing them. Her ‘inner’ point-ofaudition10 sound relates us to the narrative not as external auditors, identified with the camera and its position, nor as a participant in the dialogue, but as an internal auditor and intimate witness of the sounds Ryan can hear. Being with her and being moved with her in a visual ‘smooth’ space as an audience, we are invited to ‘active’ listening, as if we are in her situation. Anthropologist Deborah Kapchan describes such active listening ‘acts’ as enactive: “Listening acts enact – that is, they are ‘performative’, they do not simply represent sound, as waves reach the ears and are relayed to the brain, but they transduce these sound waves, changing the waves, the body and the environment in the process.” (Kapchan 2015: 36) Being isolated in a ‘space without sound’, the listening of the main character and her inner speech are crucial as anchor points. When the sound comes from ‘inside the mind of the character’, Bordwell and Thompson, speak about internal diegetic sound. They refer, for example, to Paranoid Park (2005) by Gus Van Sant where “A teenage boy fleeing from a horrific accident is plagued by inner voices, some of them auditory flashbacks. By having snatches of dialogue burst in from different stereo channels and with different degrees

52

Martine Huvenne of clarity, Van Sant presents the boy as engulfed by confusion of what to do next.” (Bordwell/Thompson 2010: 289)11

Interestingly, Bordwell and Thompson are speaking about flashbacks (about time) concerning the internal diegetic sound. They compare the sound perspective of the character with the movement of the camera as ‘a sort of sonic point of view’ (Bordwell/Thompson 2010: 292).

5 About Space This is different from Altman’s (1992: 64) spatial approach of sound in film. Starting from sound and from the lived world changes the definition of space in film and thus the filmic space. Two philosophers give us other definitions of space. They put an accent on the experience of space: For Merleau-Ponty space is not the setting (real or logical) in which things are arranged, but the means whereby the position of things become possible (Merleau-Ponty 1958: 284.) Philosopher Peter Sloterdijk puts the following question at the beginning of his “Spherology”: “Where are we if we are in the world?” He starts from the concept of ‘sphere’. “We are in an outside that carries inner worlds. (…) This place that bears the name The Sphere is the interior, disclosed, shared realm inhabited by humans (…) Living in Spheres means creating the dimension in which humans can be contained.” (Sloterdijk 2011: 27–28) Sloterdijk thus defines spaces as dynamic spheres with topological and anthropological aspects. According to him, humans create spaces (spheres) to allow them to contain their history. “The theory of the intimate set in motion with the following microsphere analysis is dedicated to showing that all human sciences have always collected contributions to a topological surrealism because it was never possible to speak of humans without having to deal with the various aimlessly wandering poetics of the inhabited interior. The spaces that humans allow to contain them have their own history – albeit a history that has never been told, and whose heroes are eo ipso not human themselves, but rather the topoi and spheres as whose function humans flourish, and from which they fall if their unfolding fails.” (Sloterdijk 2011: 90) One of his main inspirations is Gaston Bachelard’s material imagination and experienced space. For Bachelard, the experience of space, the lived space, is essentially different from the geometric space and can be evoked by

Editing Space as an Audio-Visual Composition

53

sounds or movements. I propose to take the concept of sphere as a tool in the analysis of the auditory space: through the creation of spheres (which mostly in practice are indicators as ‘layers’ of the soundtrack), different aspects of the ‘inhabited interior’ of a character can be presented through sound.

6 Editing Spaces Spatial montage is a concept that Lev Manovich (2001) uses to describe the montage of different visual images appearing on screen at the same time. He presents it as an alternative to the traditional temporal montage in which images are replaced. Starting from sound and the auditory perception of film, I argue that spatial montage in sound mostly co-exists in a temporal montage of images. Starting from the soundtrack, with the focus on the superimposition of different sound layers in relation to the space they evoke, a new paradigm of analysis of film as an audio-visual composition can be presented. Manovich takes Vertov’s Man with a Movie Camera (1929) as his guide for the language of new media focusing on Vertov’s dataset. With some shots he introduces the particular principles of new media, as for example the possibilities of montage, the combination of different spaces with each a separate identity, not merging in one space or one universe, the superhuman vision of the camera taking close-ups of any object, the universal equality of things, the spatial montage replacing the traditional sequential mode with a spatial one, etc. (Manovich 2001: XIV–XXXVI). According to Manovich, film theories from Eisenstein to Deleuze focus on temporal rather than spatial structures in film. The consequence is that in the techniques of montage, the possibility of simultaneously co-existing images in a spatial montage has not been explored systematically. But for him, a sequential narrative of the traditional temporal montage is incompatible with the spatial narrative. He refers to European visual culture to explain the spatial narrative. Looking at Pieter Bruegel’s “Children’s Game” (1560) for example, we see different games, different situations in one and the same painting. But observing Bruegel’s “The Peasant Wedding” (1566–1569), we see a combination of two different organizations of space. Still we can look at different events, presented at the same time, but in this painting brought together in a unified perspectival space. Imagine that each of the events is sounding in this painting; again different situations are combined simultaneously: the musicians playing the bagpipe, people talking to each other, a man pouring out some wine in a can, the little sounds of the boy in the foreground licking his fingers. Different from looking at the painting from one detail to the other (and thus introducing time into this static image), a representation of the sound should ask for some choices. We can imagine that the boy in the front of the painting is concentrated on what he is eating, but is meanwhile eavesdropping on some of the conversations with the music playing in the background. At that moment, different events are taking place within a single pictorial space. Manovich (2001) argues that in spatial narrative ‘all the shots’ are accessible to the viewer at once. To combine the spatial narratives

54

Martine Huvenne

with sequential narratives, he proposes the division of the screen into different frames with a logic of addition and co-existence instead of a logic of replacement (characteristic of cinema). As a consequence, the computer screen functions as a record of memory instead of as a record of perception. (Interestingly, Manovich forgets about sound.) However, with sound, the logic of replacement and the (spatial) logic of addition and co-existence of images can be combined. With a polytopic interpretation of the polyphonic structure of the soundtrack, it is possible to accumulate different spaces in one audio-visual image. This also means that, in film, auditory space can be combined with visual space at the same moment. In the editing of sound, music, and speech, different ‘spaces’ can thus be combined and evoked. When we accept that sound space and visual space are not presented in the same way, it is possible to adapt the description of spatial montage by Lev Manovich as a montage that accumulates events and images as it progresses through its narrative, to the montage of visual and auditory spaces. But can it be argued that a listener can listen to all different layers at once? While listening, choices are made. This is the big difference between the way a microphone records and the way our ears pay attention to some sounds.

6.1 The First-Person Perspective in Listening Here, a first-person perspective comes into account: in listening, I am not able to listen with the ears of another person. The first-person perspective in perception is a phenomenological concept. Applied to listening it means that the listening of someone always starts from their own perspective, at least from their position in space. Choices made in listening are defined by the listening person who, for various reasons, is interested in specific sounds, not paying attention to others. In a first-person perspective, a unity of different auditory spaces can occur. It is the listener who brings them together. Through a first-person embodied perspective of listening, a simultaneous ‘spatial montage’ is possible without explaining how the different spaces are causally related. Interiors and exteriors can seamlessly be intertwined into one experience. I start with a clear example of this first-person perspective in audio-visual perception. In Alter Bahnhof video walk12 by Janet Cardiff and George Bures Miller (Documenta 13 2012), we are, as a spectator/listener, physically at the center and go through different visual and auditory spaces. Sometimes the spaces occur simultaneously, other times they occur one after the other. It is our memory then that changes the experience and perception of the following space. At least two spaces are mixed: our physical space walking through the old train station in Kassel, and an alternative audio-visual space presented on the iPod. In this work reality, documentary, and fiction are melted. Different moments, physical spaces, visual screen spaces, auditory spaces, virtual spaces, etc. are brought together in one experienced space. The same spot changes constantly.

Editing Space as an Audio-Visual Composition

55

Being at the spot and surrounded by sound, the body of the auditor is in the middle of the experience, in the middle of the audio-visual perception. With the combination of the mediated world on the iPod and our physical space, we are also mixing past and present: “The participants watch things unfold on the small screen but feel the presence of those events deeply because of being situated in the exact location where the footage was shot. As they follow the moving images (and try to frame them as if they were the camera operator), a strange confusion of realities occurs. In this confusion, the past and present conflate and Cardiff and Miller guide us through a meditation on memory and reveal the poignant moments of being alive and present.”13 The important thing is that even if we are physically in the same location as the location we see on screen, we feel the presence of another time. With other words: on the same spot, another experience is evoked. This questions the approach to space in audio-visual perception. There is a physical space (being in a spot), but also a visual space on the screen of the iPod. Listening to the short video online, we discover different auditory layers influencing the lived space of the participant: the voice leading you during the walk, the voices of the people in the station, mediated recorded voices of people in the station, mediated diegetic music related to the screen of the iPod. The moment we see the musicians walking into the space on screen and we follow them in the real physical space, we are ‘taken’ by the music not paying that much attention anymore to what is happening in the real physical space. We are looking at the ballerina and listening to and looking at the dog entering the visual space on screen. The voice-off takes over: “let’s continue”. She is addressing her words to us in an intimate way. She sounds nearby, intimate. She is guiding us, giving us the directions we have to take, introducing people, telling stories from the past. An older voice is sounding more distantly, evoking through his words a wartime situation. The sound of a trolley brings us close to a monument, a remembrance of the deportation of the Jewish people in this station. The sound of the turning of the pages of a book containing the photographs of those people brings us in in contact with this situation. Here, we concentrate on what happened, on those people. We pay no heed to what is happening around us. The woman’s voice telling us what happened to those people takes us further on the tour to platform 3, where the Jewish people left the station. Our perception and experience of this platform are changed through the information, and we could say that we are at the same time physically on this platform, but in our thoughts and feelings we are filled not only with the information of what happened but also with the experience of what we lived the last two minutes. It is amazing how many ‘spaces’ are brought together in this walk. Through sound, we were placed at the center of the experience as human

56

Martine Huvenne

beings, combining the physical space we are walking through and the lived space, influenced by the different experiences of the evocated other ‘reality’ of the same spot. We are going through the experience of space from the perspective of a first person to bring the story together. The first-person perspective is a concept in phenomenology to accentuate that in a perception of something each person starts from their perspective (cf. Zahavi/Gallagher 2008: 13–20), which means not only their position in space as is, for example, the case of the point of view or the point of audition in film but also the situatedness of the perceiver. Merleau-Ponty describes this very clearly: “The word “here” applied to my body does not refer to a determinate position in relation to other positions or to external coordinates, but the laying down of the first co-ordinates, the anchoring of the active body in an object, the situation of the body in face of its task.” (Merleau-Ponty [1945] 1958: 115)

6.2 The First-Person Perspective and Sympathetic Listening in Gravity It is precisely the description of this first-person perspective, bringing all the different layers of perception together, that I found in the description of the sound concept of the film Gravity (2013) by Alfonso Cuarón. “The sound comes through her (Dr. Ryan Stone, the main character in Gravity) perception of sound. You overaccentuate sound and details. When you are in the situation of isolation, you hear things differently. When you are emotional, everything is crystal clear in a different way. The film pulls you on a journey with the character.”14 Being at the center of the auditory space, the listener is floating through space, moved by the sound created from ‘inside’. In Gravity, sound is transmitted through touch rather than air: when the characters touch something, the vibrations of that touch travel to their ears and so a muffled representation of the sound, is audible. “The whole contact with sound is through touch – but also through the person. Every time something’s banging against something, and she’s not touching what it’s banging against, you don’t hear it.” (Freemantle 2013)15 The above mentioned combined with the fact that the characters are floating in space without a visual horizon evokes a fundamentally different starting point to construct the auditory filmic space. The audience is listening mostly from the standpoint of Ryan. Some of her gestures are made audible in a muffled way. It is also from her perspective

Editing Space as an Audio-Visual Composition

57

that one listens to the voices in the space station, and to the voices of the other astronauts. And this perspective is not only restricted to the ‘position’ of the character in relation to the environment but also defines the interpretation of some sounds. Through sound the lived world of Ryan is built up. She is in a mediated contact with different ‘outer’ worlds: her co-astronaut Matt, the different space stations, Aningaaq on Earth. With her we are aware of her gestures. And through sound and speech, we are in contact with her ‘kinesphere’16 (her breathing and heartbeat) and her inner thoughts. These are different spaces that are built up through sound. At the same time, the audience keeps its first-person perspective, which is not always dictated by the characters. Sometimes there is a distance and an overview. So, to understand the editing of spaces in Gravity starting from the auditory side, we have to focus as much on listening and movement evoking space as on space itself.

7 Gravity In the following, I will analyze some fragments of the film to explain how different auditory spaces are edited. There is, for example, a building up to a moment in which we totally enter the ‘inner world’ of Dr. Ryan Stone. She is hallucinating. After that moment, it is also through sound that her contact with the external world is established little by little, which enables her to return to Earth. At the beginning of the film, our position towards the visual filmic space is very different from our position in the auditory filmic space. Disconnected from the visual source, we listen to voices and hear radio signals introducing the presence of human beings in an auditory way. The voices approach, words become understandable and at a certain moment we identify a woman’s voice and breathing. Information is exchanged between Houston station and the astronaut, Dr. Ryan Stone. The image shows the earth and a spaceship that is flying toward us. From that moment on, we listen with her and the center of our listening is defined through her body. Difference is made between Stone’s voice and other radio voices and radio signals in the background. One sound underlines the gliding movement of the camera, bringing the spaceship and later a human body into sight. In the beginning of the film, a male voice enters the dialogue, and it is only 50 seconds later that the first synchronized moment occurs: we see Ryan’s co-astronaut Matt Kowalski speaking. Ryan and Matt talk to each other in a mediated way. We look at them from a different perspective than when we listen with and to them. We also hear the sounds in a mediated way, but at the same time, we are situated in the auditory space near Ryan. Thinking in terms of Sloterdijk’s concept of spheres, we can describe a mediated inter-human sphere between the Houston station and the astronauts and a mediated inter-human sphere between Matt and Ryan. The ‘outer visible space’ is not audible. But rather than trying to analyze the spatial construction in this film, it is more interesting to investigate our experience of auditory space: How is the lived space of Ryan constructed and transmitted to the audience?

58

Martine Huvenne

It is clear that the ‘kinesphere of Ryan’ is at the center of the ‘experienced auditory space’ in this film fragment (see the graphical illustration in Figure  3.1). In this sphere, her voice, her breathing, and some of her gestures (when  she touches something) are audible. As a public we are in touch with her. But through the voices other ‘interhuman spheres’ are created: an inter-human sphere between Matt and Ryan, and a less personal inter-human sphere between the radio station, Matt, and Ryan.

Figure 3.1 Graphical illustration of the ‘kinesphere of Ryan’.

Editing Space as an Audio-Visual Composition

59

The experience of space is thus defined through the way the characters are in touch with each other and in contact with the space station. Although the music is paramount for the film experience and the experience of space of the audience I just mention the music without analyzing it for the reason that the music is not ‘heard’ by the characters. As there is no sound in space, the concept was designed to think about the transmission of vibrations through touch instead of air: so when the characters touch something, this vibration will travel to their ears and so a muffle representation of the sound is audible. In this way a more intimate personal (sphere) is created, accentuating some of Ryan’s gestures. When she later enters the space capsule, those gestures will sound different since then sounds resonate through her environment, which will then create another ‘intimacy’ with her. When Houston announces that the mission is aborted because of the flying debris and that the astronauts immediately have to return to Earth, the soundtrack changes. At the moment we see the debris, sounds are moving through space, the music is moving through space, and the voices are moving with the characters who, in their turn, are moving through space. As an audience, we don’t have any direction in the auditory space. The disorientation is at a climax. When the contact with Houston is lost, and Stone’s voice is off screen, we keep on hearing her voice and breathing. She is not able to define her position. The music takes over to express the panic. Two minutes later, we enter in an entirely different auditory space: we listen from the inside of her helmet. She calls Kowalski. He doesn’t answer. Her voice is not mediated anymore. We entered her kinesphere. She is drifting, and we are drifting with her until we leave her kinesphere and listen to her from a distance. Kowalski is contacting her. Later, when Ryan is in need of oxygen, we enter into her state. Music underlines this change of state. We still hear the music and the conversation between Kowalsky and Ryan. But she interiorizes, talking about her daughter who died when she was four years old in a senseless accident. Kowalski decides to detach and floats away to give her the possibility to enter the Soyuz spaceship. At this moment, we live her situation, her gestures, and some music. As an audience, we are situated in her kinesphere: We still hear Kowalski talking to her. We hear radio signals, her breathing, her voice, the muffled sound of her gestures and music. When Ryan wants to return to Earth but discovers that there is not enough fuel, we see her through a window of the Soyuz. She is trying to connect with a space station. We first hear her voice from far off, then from nearby. At a certain moment, we hear her breathing and the other voice saying “Aningaaq”. She is in contact with someone, but there is confusion: they are speaking a different language and do not understand each other. What happens then between Ryan and Aningaaq is very interesting: although they are connected, they remain in their loneliness and their personal ‘sphere’. Both personal spheres interfere through the sounds. But more and more, Ryan enters an inner world, not

60

Martine Huvenne

reacting anymore to the sounds she is hearing. She thinks of her lost child and falls into a sleep modus. This introduces her hallucination: Matt enters her cabin and tells her how to escape from this situation.17 Starting from the concept of ‘auditory spaces’, we can describe this scene as given in Figure 3.2.

At first Ryan is totally isolated in her cabin. We hear her breathing and gestures (her kinesphere).

She tries to connect with her voice: “My name is Ryan Stone, I need help” (her voice is broken). We hear the voice of Aningaaq talking to her. An inter-human sphere between them is installed, but no real communication takes place.

She listens receptively to the environment of Aningaaq on Earth, where he lives. Dogs are barking.

She tries to imitate the dogs. Aningaaq also begins to imitate the dogs barking, which gives both of them a real human contact, through their voices, without words.

Editing Space as an Audio-Visual Composition  61

Ryan begins to disconnect, turning into herself, although she is still connected with Aningaaq. She is talking to herself: “I’m dying, Aningaaq”.

In his ‘inter-human sphere’ and his environment, Aningaaq speaks to another person and sings a lullaby for his baby; Ryan is listening to the lullaby from a distance. Again she is listening receptively, but it touches her very deeply.

The sound of the lullaby brings her into an inner world.

The lullaby evokes the memory of her own child who died at the age of four. She asks Aningaaq to keep on singing to make her fall asleep.

Figure 3.2  Graphical illustration of the two universes combined in the scene from Gravity.

62

Martine Huvenne

In this part, two universes are combined: the world of Aningaaq on Earth and her isolated, personal (inner) world. Aningaaq is also isolated, living in an extremely silent spot. However, he is surrounded by his dogs, his wife, and his child. Every sound is connected with his lived space. Happy to be in contact with someone else through the radio contact, he continues to talk even when he doesn’t understand the other voice.18 From his side he tries to talk to her, then they are laughing together. When he is also imitating the dogs, they enter into the same ‘sphere’. This leads to the moment when he is singing to his baby, she is listening to him and integrates his voice and his lullaby into her inner universe.

8 Conclusion To analyze the ‘editing of spaces’ in Gravity, I started from some fundamental questions in the spherology by Peter Sloterdijk and Merleau-Ponty’s vision of experienced space. With this approach, I propose a spatial analysis complementary to the spatial analysis that starts from the image on screen (on-screen/off-screen). I kept the distinction between diegetic and non-diegetic space and between external diegetic and internal diegetic space (Bordwell/Thompson 2010), but I proposed a detailed elaboration of the (internal and external) diegetic space. Informed by the sound concept, an analysis of the lived space of Dr. Ryan Stone was evident. The concept of the director, the sound designer, and sound supervisor invites the audience to move with the character (moving with the first-person perspective of the character). Through the composition of different ‘spheres’ (lived spaces), the different ‘experiential worlds’ of Stone are unified in one experience through the audience. With this proposition I applied Merleau-Ponty’s ([1945] 1958: 281–347) embodied approach to space to film analysis: Space is no longer described and defined from an external standpoint, an overview, but as something that is constituted through experience.

Notes 1. See also http://www.dcinematoday.com/dc/pr.aspx?newsID=3392 [last accessed: 01 March 2016]. 2. This was presented by Miguel Mera at the MaMi conference 2015 in New York: “Towards 3-D Sound: Spatial Presence and The Space Vacuum”. 3. Source: https://vimeo.com/76123849 [last accessed: 01 March 2016]. 4. http://www.theverge.com/2013/10/10/4822482/the-sound-design-of-gravityglenn-freemantle-skip-lievsay [last accessed: 01 March 2016]. 5. This refers to Rick Altman’s concept of spatial signature: “[…] for what the record contains is not the sound event as such but a record of particular hearing, a specific version of the story of the sound event. Every recording is thus signed, as it were with the mark of particular circumstances in which it was heard […]. Every recording carries the elements of this spatial signature, carried in the

Editing Space as an Audio-Visual Composition

6. 7. 8. 9. 10.

11.

12. 13. 14. 15. 16. 17. 18.

63

audible signs of each hearing’s particularities. […] even when they seem not to match the visual data provided with the sound record, they still carry information that is narrative and spatial in nature” (Altman 1992: 24). ‘here’ is understood in the way Merleau-Ponty describes the ‘here’ of a phenomenological body in a lived space, see Merleau-Ponty [1945] 1958: 115. http://www.theverge.com/2013/10/10/4822482/the-sound-design-of-gravityglenn-freemantle-skip-lievsay [last accessed: 01 March 2016]. See http://www.dcinematoday.com/dc/pr.aspx?newsID=3392 [last accessed: 01 March 2016]. See http://filomasnou.blogspot.de/2012/08/haptic-touching-film-object.html. [last accessed: 1 March 2016]. Point-of-audition sound always carries signs of its own fictional audition. As such, point-of-audition sound has always the effect of luring the listener into the diegesis not at the point of enunciation of the sound, but at the point of the audition. Point-of-audition sound thus relates us to the narrative not as external auditors, identified with the camera and its position, nor as a participant in the dialogue, but as an internal auditor. We are asked not to hear, but to identify with someone who will hear for us. Instead of giving us the freedom to move about the film’s space at will, this technique locates us in a very specific place – the body of the character who hears for us (Altman 1992: 60). A less technical elaborated but similar example can be found in Psycho (1960) by Alfred Hitchcock when Marion sits in her car. Saying ‘hello’ when her boss passes by, she realizes that he can discover that she is running away with the sum of money she had to bring to the bank. Cardiff & Miller Video walk, 26 minute walk. Produced for Documenta (13), Kassel, Germany. The video is available at: http://www.cardiffmiller.com/artworks/ walks/alterbahnhof_video.html [last accessed: 01 March 2016]. See http://www.cardiffmiller.com/artworks/walks/bahnhof.html [last accessed: 01 March 2016]. See http://www.theverge.com/2013/10/10/4822482/the-sound-design-of-gravityglenn-freemantle-skip-lievsay [last accessed: 01 March 2016]. http://www.theverge.com/2013/10/10/4822482/the-sound-design-of-gravityglenn-freemantle-skip-lievsay. [last accessed: 01 March 2016]. The kinesphere is the sphere around the body whose periphery can be reached by easily extended limbs without stepping away from that place which is the point of support when standing on one foot (cf. Laban 1988: 35). See https://www.youtube.com/watch?v=JaFI6hGm3fI. [last accessed: 1 March 2016]. Jonas Cuarón, son of Alfonso Cuarón, worked out this situation in a short film in which sound and image are a function of each other. What is happening in sound in this film is a function of the story of a fisherman in Greenland, isolated from the rest of the world with his wife, child, and dogs. His only way to be in contact with the world is the space station.

Bibliography Altman, R. (2008). A Theory of Narrative. New York: Columbia University Press. Altman, R. (1992). “Sound space”. In: Altman, R. (ed.). Sound Theory, Sound Practice. New York: Routledge, pp. 171–177.

64

Martine Huvenne

Altman, R. (1992). Sound Theory, Sound Practice. New York: Routledge. Bischop, B. (2013). How the Sound Masters of ‘Gravity’ Broke the Rules to Make Noise in a  Vacuum: Sound Designer Glenn Freemantle and Re-Recording Mixer Skip Lievsay Reveal the Film’s Sonic Secret. Online: http://www.theverge. com/2013/10/10/4822482/the-sound-design-of-gravity-glenn-freemantle-skiplievsay [last accessed: 01 March 2016]. Bordwell, D./Thompson, K. (2010). Film Art: An Introduction. 9th edition. New York: McGraw-Hill. Boulez, P. (1971). Boulez on music today. London: Faber and Faber. Bull, M./Back, L. (2003). The Auditory Culture Reader. Oxford: Berg. Cardiff, J./Miller, G. (2012). Alter Bahnhof Video Walk. Online: http://www. cardiffmiller.com/artworks/walks/bahnhof.html [last accessed: 01 March 2016]. Chion, M. (2009 [2003]). Film, a Sound Art. New York: Columbia University Press. Coleman, M. (2013). Soundworks Collection: The Sound of Gravity. Online: https:// vimeo.com/76123849 [last accessed: 01 March 2016]. Dolby Atmos. (2013). Online: http://www.dcinematoday.com/dc/pr.aspx?newsID=3392 [last accessed: 01 March 2016]. Eisenberg, S.J. (2015). “Space”. In: Novak, D./Sakakeeny, D. (eds.). Keywords in Sound. Durham/London: Duke University Press, pp. 193–207. Gallagher, S./Zahavi, D. (2008). The Phenomenological Mind: An Introduction to Philosophy of Mind and Cognitive Science. London: Routlegde. Grant, G. (2012). (Haptic). Touching the Filmobject. Online: http://filomasnou. blogspot.de/2012/08/haptic-touching-film-object.html. [last accessed: 1 March 2016]. Huvenne, M. (2012). Het geluid als een innerlijke beweging in de overdracht van een ervaring in film: een fenomenologische benadering. Amsterdam: Vossius Press Uva. Ihde, D. (2007). Listening and Voice: Phenomenologies of Sound. New York: State University of New York Press. Kapchan, D. (2015). “Body”. In: Novak, D./Sakakeeny, M. (eds.). Keywords in Sound. Durham/London: Duke University Press. Manovich, L. (2001). The Language of New Media. Cambridge/London: MIT. Marks, L. (2000). The Skin of the Film: Intercultural Cinema, Embodiment and the Senses. Durham/London: Duke University Press. Merleau-Ponty, M. (1958). Phenomenology of Perception. London/New York: Routledge. Parker, R. (2014). Gravity – Clip: Aningaaq’s Lullaby. Online: https://www.youtube. com/watch?v=JaFI6hGm3fI [last accessed: 1 March 2016]. Robertson, R. (2011). Eisenstein on the Audiovisual: The Montage of Music, Image and Sound in Cinema. London: I.B. Tauris. Sloterdijk, P. (2011). Bubbles. South Pasadena: Semiotext(e). Sloterdijk, P. (2011). Bubbles. Spheres 1. South Pasadena: Semiotext(e). Sobchack, V. (2004). Breadcrumbs in the Forest: Three Meditations on Being Lost in Space. In: Sobchack, V. (ed.). Carnal Thoughts: Embodiment and Moving Image Culture. London: University of California Press, pp. 13–35. Sobchack, V. (1992). The Address of the Eye: A Phenomenology of Film Experience. Princeton: Princeton University Press.

Editing Space as an Audio-Visual Composition

Filmography Alter Bahnhof Video Walk (2012). Cardiff & Miller. Germany. Gravity (2013). Alfonso Cuarón. Warner Brothers. UK/USA. Man with a Movie Camera (1929). Dziga Vertov. VUFKU. Soviet Union. Paranoid Park (2005). Gus van Sant. MK2 Productions. France/USA. Psycho (1960). Alfred Hitchcock. Shamley Productions. USA.

65

4

Movie Physics or Dynamic Patterns as the Skeleton of Movies Wolfgang Wildgen

1 Introduction: Is There Something to Explain in Movies? If a spectator comes home from cinema, he/she may revisit the story, the sequence of major events, spectacular action scenes, the climax, and the (happy) end. The major characters, the beauty/sexiness of the female protagonists, the cleverness of the hero or the villain may come to mind. But is he or she keen to understand, let alone be able to explain, the picture? Probably not. Who is then the addressee of an analysis of films? A student of cinema, who is preparing for their activities in the field, either as cameraperson, script-writer or director, may ask more specific questions. Truffaut, who started as a film critic and later produced his films, asks the question: Mr. Hitchcock, how did you do this? In his interview with Hitchcock in 1966 (cf. Truffaut 1968), he is particularly interested in the question: How to express oneself with purely visual means? What counts is not the story which may derive from some novel or newspaper report, but the visually offered information and emotions. Contrary to novels, a movie does not primarily tell us what has been spoken, said, or told by the protagonists. The minor relevance of language for movies had previously been demonstrated in the era of silent pictures and Hitchcock and Truffaut argue in the interview in favor of silent pictures: “Hitchcock: Well, the silent pictures were the purest form of cinema; […] Truffaut: In this sense one might say that mediocrity came back into its own with the advent of sound. Hitchcock: I agree absolutely […]. When we tell a story in cinema, we should resort to dialogue only when it’s impossible to do otherwise.” (Truffaut 1968: 49) In a sense, sound as noise in the environment and as language spoken is just an index of our acoustic environment; it may reinforce the visual impact, but cannot replace what we see with our eyes. What is important is the attention of viewers, which must be guaranteed at any time, and this attention is mainly visual. Linguistic information may even compromise this attention and the effects of surprise and suspense, which are central to the success of a picture.1

Movie Physics or Dynamic Patterns as the Skeleton of Movies

67

Beyond bodily action, dialogue may also be relevant as a kind of social action. It is simultaneously a visual process, which includes gazing, posture, and other paralinguistic cues. Fundamental filmic techniques including utilization of such cues had already been formed in the era of the silent picture, mainly with and after Griffith. The position and mobility of the camera, i.e. the filmic viewpoint, is the major difference to the theater where the classical proscenium viewpoint dominates. By the use of different cameras and by montage, the spectator can change his/her position in relation to the reported actions. He or she can adopt subjective or objective viewpoints. Hitchcock asks here for “realistic viewpoints”, i.e. those which a human viewer can normally take or imagine taking (Truffaut 1968: 77). If a close-up is necessary, he introduces (in Rear Window (1954)) the telescopic lenses of the main figure Jefferies (James Stewart) as a natural cause for the change of focus. In general, he respects the natural affordances of the environment. Thus, Jefferies is by his profession press-photograph and in his situation of immobility, he uses his normal instruments: lenses, cameras (with flash in the defense scene), and the telephone. His observation through the window corresponds to normal human behavior in similar situations (“We are all a race of Peeping Toms”). Thus, natural observation, curiosity, the interpretation of body cues, postures, faces, eye glances, and gestures are transferred into techniques of the camera and the construction of shots. But this is not enough. As soon as films became longer and took over a narrative function, thus rivaling language and literature,2 this demanded a specific economy of visual display. This had three major consequences: 1. Pieces of visual information had to be packed and later reassembled into a plot by montage. The technique of cuts, i.e. small structural sequences, which correspond in their duration more or less to sentences (5 to 15 seconds; cf. Truffaut 1968: 148), may have been copied from linguistic codes. Or they also correspond to the more general cognitive principles of economy in relation to short time memory and closure enabling larger constructions (texts, discourses, etc.).3 Larger pieces may constitute “little films in themselves” as Hitchcock points out to Truffaut in the interview (Truffaut 1968: 75). In Rope (1948), Hitchcock tries to match the picture time to the event time (7:30 to 9:15), and he uses the maximal length, i.e. the 10 minutes of a film reel. To maximize the impression of time continuity, Hitchcock even applies a trick: one reel ends with a close-up on the black jacket of someone, while the camera team is changing to a new reel, which starts with the same black surface.4 2. Montage is the technique used to reassemble scenes. Hitchcock had already had good experience with montage in the era of silent pictures before he became a film-director. He worked on titles in silent pictures. With the choice of titles (captions often accompanied by a drawing), the story of the silent picture could be modified, even from a tragedy to a

68 Wolfgang Wildgen comedy and vice-versa and bad performances of actors could be made (more) acceptable (cf. Truffaut, 1968: 24–25). Later, the way he had conceived and realized the pieces of the picture were such that he was the only one who could correctly assemble them. In a certain sense, no one else even knew how the whole story would look. This guaranteed that the producer or other authorities could not freely use his material to produce a picture which did not map with his plan and purpose. The assembling by montage does not just produce a serial order out of a heap of film materials; it follows on from a kind of plan or diagram. The storyboard, of which Hitchcock had training as an industrial designer, visually included camera views of important moments and the basic sequence of shots. As also suggested by Cohn (this volume), it is akin to a comic strip. Hitchcock used illustrated storyboards and for the movie Family Plot (1976) he even employed a “Production Illustrator” (cf. Spoto, 1976: 463–499 for a comic-like storyboard). 3. Normally, movie time has only a very indirect relation to real time. Flashbacks may even reverse the direction of time. It is not an easy task to visually mark the difference between ongoing action time (even if it is presented as a sequence of temporal fragments) and past episodes rehearsed in memory or even with dream episodes. Propositional attitudes such as: ‘He said’, ‘I thought’, etc. cannot directly be depicted visually. Timing is also vital for the clarity of understanding from the perspective of the audience. Thus, a newly introduced element or piece of an event must be properly prepared. In a famous scene of North by Northwest (Hitchcock 1959), Gary Grant stands lost in a bare landscape, without dialogue and music. Eventually, when a car, arrives and a farmer begins to wait for his bus on the other side of the road, a short dialogue introduces the topic of the crop plane, which suddenly shows up and tries to kill the protagonist.5 In this scene, the pace of visual information is adapted to the velocity of the spectator’s understanding. When the plane attacks, Grant escapes and finally hides in a cornfield. In this scene, the timing corresponds to a realistic lapse of time in relation to the distance and speed of locomotion. Thus, the dynamics of motion and action are the structural anchor, whereas the psychological timing reflects the cognitive processes in the audience. These general remarks could make us believe that pictures are the product of intricate constructions, and finally, movies could just be fakes, as the film f for fake by Orson Wells (1973) suggests. In contrast to this, I intend to portray here how movies have an inevitable realistic grounding, which is also necessary for the understanding of very fictional, even fantastic plots. I  will call this foundational level movie physics. Hitchcock used the term of “mechanics of film”, referring to the movie Secret Agent (Truffaut 1968: 185).6 In section 2, I will discuss the relation of movie physics to

Movie Physics or Dynamic Patterns as the Skeleton of Movies

69

semiophysics (as proposed by René Thom), to the kinematics and dynamics of action and the question of realism in movies. Section 5 will focus on action movies with a special emphasis on action in James-Bond movies.

2 Physical Dynamics and Movie Physics7 Dynamics (and kinematics) lies at the heart of modern natural sciences: Galileo laid the fundamentals of kinematics, Kepler stated his laws of planetary motion, and Newton formulated the basic principles of modern physics (in 1687) which were understood as universally valid until Einstein published his ideas on special relativity in 1905. Galileo’s relativity principle states that velocity is relative to an inertial system. Acceleration and the forces (counter forces) which govern it are the dynamic reality with which we are concerned.8 Classical examples of simple dynamic systems are pendulums, which have been used technically since Huygens’ invention of the pendulum clock in 1656. More complicated are pendulums in a series which transmit force to one another, coupled pendulums, and double pendulums.

Figure 4.1 Pendulums in a series of coupled pendulums and the light trace of a double pendulum.

The technical models in Figure 4.1 give an idea of basic dynamical systems and effects like the transmission of force and forms of coordination/ resonance and turbulence (dynamic chaos). Such simple situations become very complex if the balls interact in two dimensions (billiard ball case), if different strength of links exists between pendulums (coordination between partners of different weight) or if the double (triple) pendulum moves in three dimensions. In the domain of living agents, forces alike to those of billiard balls are exchanged by athletes in boxing. Coupled pendulums appear if two or four limbs are coordinated in locomotion or in social coordination (building of conventions, rules). Arms and legs are basically double pendulums (with restricted freedom). We shall apply these basic concepts in later sections of the chapter. In the 80’s of the last century, René Thom introduced the dynamic way of thinking into semiotics (and linguistics, science of history, etc.).9

70 Wolfgang Wildgen Mathematics (vector analysis, differential equations, catastrophe theory, etc.), which is successful in dealing with dynamics in nature, is then applied to model very basic situations in social interaction and communication.10 The term movie physics is to be understood in this tradition of thought: forces, counter forces, acceleration/deceleration, effects of forces, the timing of motion and action in moving contexts are put into the center of film analysis. This does not mean that film becomes a proper field of research for physicists; rather that language, film, and media share universal principles which show up in nature and culture. At its heart lies the concept of motion, acceleration, and conflict in space, its perception, memorization, categorization and its representation in sign structures. The evolution of animals can explain the specific link between physics and semiotics with vision, audition, and brains. In order to survive in a world governed by physical laws, animals and humans had to adapt not only their bodily constitution (e.g. mobility) but also their senses (vision, audition, etc.) and, in the sequel, their means of cognition and communication such that their locomotion and action in space/time responded adequately to the physical laws governing their environment.11 This is not the place to elaborate the theoretical aspects of this semiotic approach, which defines the line of research called dynamic semiotics (cf. the contributions in Wildgen/Brandt, 2010). I shall rather show its impact in a set of applications mainly to action films, because here motion, chase, and flight, fight and challenge dominate the narrative content.12

2.1 Dynamics In Language and Film In written language, dynamics are restricted to the sequence of units (graphemes, morphemes, sentences, chapters, etc.). In spoken usage, intonation, rhythm, and emphasis are additional dynamic features. In the case of photos and even more clearly in film, the space depicted is two-dimensional (with a simulated third dimension). However, in classical structuralism, spatial and dynamical aspects are reduced to discrete choices and concatenations. Thus, Metz (1968) argued within the framework of duality: paradigmatic versus syntagmatic, established by Jakobson and others. Each point step in a process is a choice made from a list of alternatives (the paradigm); the respective choices are structured under the syntagmatic restrictions of the semiotic system. This may be further reduced to the operation of a machine controlled by a program. Film, however, produces a continuous, multi-dimensional perceptual space and therefore does not fit this simple picture.

2.2 Film and Photography The history of film is linked to the earlier technique of photography. As Souriau (1969: 126) shows, both film and the photo refer to luminosity

Movie Physics or Dynamic Patterns as the Skeleton of Movies

71

(“éclairages, projections lumineuses”). Now, many arts refer to visual effects, and thus to light (in Souriau’s table: drawing, sculpture, architecture, painting, dance), but in the case of film, the effects are transitory and the projection does not change the surface on which the film is projected. The visual impressions are momentary and thus transient in time (including very elusive memory effects). In this respect, film may be compared to dance and theater. In its early history, cinematographic techniques belonged to circuses or amusement parks and were on a par with magical performances. Their narrative potential was only developed in the period from 1910 to 1915. The generation of Méliès, Porter, and Griffith invented the cinematographic code for filmic narratives. It was only in this period that film acquired what Metz calls a ‘language of film’, primarily due to the intention of fulfilling narrative functions (thus rivaling novels and literature). In a certain sense, the narrative function is a freeloader on the visual medium film; and movie-art consists mainly in expressing content in an immediate and natural visual technique and not just as an illustration of a story told by linguistic means. The focus of visual media is then on the referential function, i.e. on actions. They may be decomposed into actions of protagonists and antagonists. Chases drive the complication phases. Several complicating actions are either ordered serially or in parallel sequences. Action can be accelerated or decelerated. In extreme cases, the movie may either simulate a steady flow or dissolve into a tachistoscope-like sequence of cuts, although normal films consumed by a large public avoid these extremes (cf. Biro 2008).

2.3 The Semiotic Construction of Space in Film The space of the medium is two-dimensional and contains a time dimension. A film in the Academy format (rectangular 11:8) emphasizes the center and reinforces the illusion of depth. The broadband format (11:5) emphasizes the horizontal dimension, such that landscape and action scenarios become more prominent. Actions and movements in the horizontal dimension of the filmic space can be tracked for a longer period without a change of focus. A third dimension of space may be simulated, either by motion into depth or via 3D effects. The direction of view of the person (and the camera) can go down from a balcony, a window in the upper floor into the courtyard, the street, or up into a stairwell or, in a particularly extreme case, such as rock climbing, it can go into a vertiginous abyss. An example of a movie focusing on the vertical perspective is Vertigo (Hitchcock 1958), in which the perspective down from the bell tower or into the staircase is a dominating topic (see Figure 4.2a). In North by Northwest (Hitchcock 1959), the protagonist and his female partner climb Mount Rushmore (below the presidential head figures) and he must rescue her when she is in danger of falling off the cliff (see Figure 4.2b).

72 Wolfgang Wildgen

Figure 4.2 (a) Screenshot from Vertigo (Hitchcock 1958) and (b) Screenshot from North by Northwest (Hitchcock 1959); look into the abyss.

Major types of space constructions in movies are: a) landscapes and locations which are carefully selected or reconstructed; b) private rooms: they are used to illustrate the properties of the agents and their spaces of power, they also represent power relations between groups and individuals; c) transitional spaces (passage, transit): in scenes on streets, hotels, trains, airports, and so on, characteristic paths of actions and the physical and emotional forces operated are visible; d) maze-like spaces: actors and audience lose orientation in a given space or they only exit safely with difficulty; e) artificial areas (e.g. medially constructed ones) and virtual spaces that are digitally created. If the movie involves architectural interiors, elements of architecture, such as doors, staircases, windows, narrow corridors, room dividers, and even furniture, can create their framework, thereby structuring the space. Persons can be assigned individual space sectors. These spatial divisions can be repeated when agents move through a suite of rooms. I illustrate this topic by mentioning some specific experiments with movie spaces. The Danish directors Lars von Trier and Thomas Vinterberg set out the rules of Dogme 95, which ask for a return to natural conditions of observation via a camera and forbid all kinds of illusionistic manipulations. The film, Dogville (von Trier 2003) takes this minimalist program to its extreme. The architecture (walls, doors, etc.) is represented by categories (as in language) and not via spatial illusions. The viewer looks through the walls. The abstract scenario can be seen as an antithesis to film technology and fake-construction (see Figure 4.3a). Another film which highlights the phony portrayal of space construction in movies and television is The Truman Show (Weir 1998). The basic events occur in an illusory world, whereby the protagonist is eventually able to see and touch the constraining wall and finally exit into the real world (see Figure 4.3b). The agent must discover the degree of falseness while the spectator guesses it early in the

Movie Physics or Dynamic Patterns as the Skeleton of Movies

73

Figure 4.3 (a) Screenshot from Dogville and (b) Screenshot from The Truman Show.

picture and is better informed than the hero. This produces a type of suspense (see also next section). In expressionistic films, e.g. Metropolis by Fritz Lang (1927), the city as labyrinth may be a topic. The megacity in Metropolis is huge and shows a large population in steady motion. It is subdivided into two classes: those living above and those living below. Human beings move like insects in this huge urban construction. Human action appears meaningless if it is not a revolt against these conditions of life. Production designers often have to make a great effort to create a place that is suitable for the illusion of a film. This effort is justified because the location of the action is the anchor for all actions and makes them appear credible. In addition, characters and actions are only effective in the context of these places. In a broader sense, this includes costumes and the sitespecific behavior of the characters. In this respect, the basic structure of the film is already given by the construction of places and the control of the events in these places. In connection with the planning of space, actions may receive specific meanings. The structuring of space, especially by the effect of separating lines and thresholds, produces meaning since it creates bonding structures between thematically related partial fields that are spatially separated.13 In films, the transformation of spatial structures is added to the movement of people and the camera. One can even view the movie as the medium of space transformation; see also Prange (2012) for the history of filmic space constructions. The surrounding space can flow past the moving person as that person is being focused in the foreground. This is particularly evident in older Hollywood films, and some films of the Nouvelle Vague, where overly long shots of the actors are filmed at the wheel of a car and the driving motion is inserted via an independently filmed background. But movement may also result from the fact that the camera is moving, or via a change of camera focus from a long shot to a close-up.14 In the construction of meaning, the cameraperson substantially contributes (instructed by the director) to the work. The sequence of motion scenes and actions in different spatial segments is done by montage (in the editing room).

74 Wolfgang Wildgen Space is not just a static background for events and actions. It allows for specific configurations of action, motion patterns, and accidents. This is clearly the case for means of transportation. The actions play out in or on these spatial domains; but these places are also instruments for the action, i.e., their movement can be used for actions or can hinder them. The mastery of space and the laws of motion in space distinguish the hero and are the reason for his or her superiority and victory at the end of the film.

2.4 Realism in Film and the Physics of Action The following is an old question in the aesthetics of fiction and theater: How realistic should the story be to persuade the reader/public that the content is relevant in the world in which we live and to avoid the impression of unnaturalness, arbitrary construal, and lies?15 At the end of the 19th century, the triumph of photography and its use in the context of arts together with the literary projects of realism and naturalism triggered a new wave of realism in theater. The cinema continued and reinforced this trend; the aim was to make the audience forget that it is watching a film and to eliminate all elements which show that the film is an artifact produced via a camera and a projector.16 The central question of realism in films concerns motion and interaction in space. One can ask: Is the scene physically possible given the laws of gravitation, free fall, locomotion, and possible effects of forces (jumps, blows, pistols, and moving projectiles)?17 In thrillers, the tendency is to approach the limits of probability to demonstrate the excellence of the hero and to a lesser extent that of his or her enemies.18 In many instances, one can doubt the physical realism of such scenes. In the film Goldfinger (Hamilton 1964), Bond’s adversary is sucked out of a plane after his bullet has destroyed a window, which can be shown to be physically impossible. In the film Casino Royale (Campbell 2006), Bond makes a Bungee jump from the height of a barrage. Tolan/Stolze (2008) show that the duration of 13 seconds is unrealistic given the distance of 100m, the acceleration of a falling body (Bond), the elasticity of the bungee rope, etc. The film-makers had to use a slow motion picture and to repeat elements of the jump from different angles to arrive at 13 seconds from the realistic value of 4.5 seconds. Many scenes are, however, at the limit of realism. Thus in the film Moonraker (Gilbert 1979), Bond is thrown out of a plane by Jaws and tries to reach the pilot, who sprang three seconds earlier, and is descending with a parachute. Jaws, who springs 33 seconds after the pilot, is able to reach Bond, who is now hanging on his parachute after he has successfully fought the pilot. Tolan/Stolze (2008) calculate that the plane must have been at least 6000 meters above the ground. Under this condition, both events could, at the limit, be possible (the plane was probably only 3000m from the ground). In Casino Royale, the chase between Bond and Molakka is also at the limit of credibility because some of the jumps are so deep that the

Movie Physics or Dynamic Patterns as the Skeleton of Movies

75

forces which need to be absorbed by the body at the landing are extreme. Tolan/Stolze, (2008: 30) calculate a force of 16400 Joule, i.e. 1.6 tons (the stuntmen jump into a package of cardboard boxes). In Casino Royale, Bond runs after his adversary during a long series of shots. If we consider their speed, we can guess that they would be exhausted after ten seconds. In a similar vein, horse chases in typical Western movies would exhaust the horses after one mile. The question is, however: Is physical realism relevant for the audience? Realism, in the eyes of the viewer, is phenomenological in its nature, i.e. he/she can imagine the action (if he/she has performed similar actions him/ herself) and experience the force he/she has to use to accomplish the action or to absorb the energy of a jump of a certain depth. Judging by this, he/she can use extreme experiences such as witnessing a fabulous world record in running or jumping. In their judgment, the viewer uses a kind of tolerance space, i.e. values in the neighborhood of what is known to be possible are realistic. This version of tolerance realism based on phenomenological experience will be assumed in the following analyses. We may distinguish different levels: 1. Realism in terms of perception. The iconicity of a photo or a film shot is adjusted to human perception and its automatisms. 2. Process-realism. When understanding the image or the movie-shot context-dependent disambiguation is unconsciously provided. 3. Content realism. A selection (not the totality) of realistic scenes is chosen. The selection and reframing add a momentum of construal to it (this is typical of documentaries or reports based on documentary materials). Physical realism is the baseline; the three levels mentioned above and tolerance spaces on each level then define a proper space of realism in image and movie.

3 Basic Types of Kinematics in Movies In reality and correspondingly in films (specifically in action movies), one can distinguish different positions on a scale going from states/rigid positions to very fast and complicated motions. Physical analogues of the dynamics in movies (movie physics) are: 1. The inertia of a body in space without external forces acting on it. Examples are a steady state or a totally damped oscillation. The pendulum in a viscous liquid is the physical prototype. Galileo’s inclined plane or the free fall corresponds to motion with a constant acceleration. Thus, in the case of jumps and scenes where objects or persons roll and glide down a slope they make use of these dynamics: cf. the

76 Wolfgang Wildgen famous ski chases in the James Bonds films On Her Majesty`s Secret Service (Hunt 1969) and The Spy Who Loved Me (Gilbert 1977) and the plane gliding downhill on snow in Spectre (Mendes 2015). 2. A steady motion can be combined with forces acting on it. The physical prototype is the elliptical motion of the planets attracted by the sun (see Kepler’s laws). Along with natural forces, a human can be a force which directs motion, locomotion, and action. If two humans interact, the force field may be coordinated or antagonistic, symmetrical or complementary (cf. Watzlawick’s fifth axiom of communication in Watzlawick et al. 1967). 3. With three moving bodies depending on each other, chaotic modes occur. If the bodies are very different in size, the system is relatively stable (e.g. the system of several planets moving around the sun). Human action may, however, be confronted with chaotic dynamics in cases where more than two human wills with similar force profiles interact or apply to the same object/process (cf. triangular social relations or ménage à trois). In the following, I shall discuss correlates of these motion patterns in movie physics.

3.1 Steady Motion and Lack of Motion or Rigidity This position is found in classical art, in photography, and to a limited degree in films (mostly in short passages at the beginning or the end of a story): 1. The movement of a river, passers-by, traffic, or the wind and the waves. This is the eternal movement of the type of “Panta Rhei” in Heraclitus. A character walking in the street, a car rolling on a road is experienced as (almost) forceless. Biro (2008) distinguishes “turbulence” and “flow” in movies. “Flow” corresponds to steady motion. 2. The classic landscape or architecture picture in Fine Arts has a parallel in descriptive scenes of the film, when the camera either shows a landscape slightly in motion by wind in the branches, a town, an interior or if it moves steadily on a car, a train, etc.. 3. The static portrait has a filmic counterpart in close-ups of the face, which often have a characteristic rigidity. This can refer to hidden emotional (in a romance) or cognitive processes (in a thriller). In the closing scene of Queen Christina (Mamoulian 1933), where Christine (Greta Garbo) leaves Sweden and her throne on a ship, the immobility of her body and face are characteristic. The mechanical counterpart of this dynamic/static type is the damped pendulum, which makes the system remain close to resting point (see Figure 4.4).

Movie Physics or Dynamic Patterns as the Skeleton of Movies

77

Figure 4.4 (a) Quasi-still in the final scene of Queen Christina with Greta Garbo and (b) pendulum with rest position.

3.2 Motion Controlled by a Force In a pendulum, the original motion pushing it refers to a specific force in the system (e.g. clockwork) or its context; the rest position is a minimum position, i.e. either the forces disappear, or contrary forces nullify each other. In a planetary system, gravitation is the basic force which transforms the normal linear motion of the planet into a stable orbit if the distance is adequate. In the context of locomotion and action beyond physical and muscular forces, a kind of will is assumed as controlling force. Major types of controlled motion are: 1. The targeted locomotion. This has two limitations: start and finish. Often the target dominates; e.g. the extreme West in the Western film (frontier motive: the settlers try to reach the far West) or marriage in a love movie. 2. Spatially structured movements. The space has boundaries and transitions. Consecutive episodes repeat similar actions (e.g. chases) in different spatial contexts. Typically the series ends with a final scenario: the death of the antagonist, the union of the lovers. The different spaces may be linked to different shots or appear simultaneously on the screen. Burch (1983) enumerates six boundaries of the “movie cube”: onscreen: left side (1) – right side (2) – above (3) – below (4) –in front (5) – behind (6) and offscreen. Spatial transitions are therefore either represented by camera angle or motion, montage or onscreen transitions. An example of a targeted and force-controlled locomotion is given in Figure 4.5: the showdown of Bonnie and Clyde (Penn 1967) repeats the basic scenario. Characteristic phases are: driving (by car) → stop → confrontation → death. In the final scene, the protagonists (the gangsters) are killed and not their adversaries.

78 Wolfgang Wildgen

Figure 4.5 Screenshots from the final scene in Bonnie and Clyde (Penn 1967), after many episodes of crime.

3.3 A Plurality of Movements Affecting Each Other 1. Simple physical analogs are cascaded pendulums which transmit the initial momentum from the first to the last one in a chain. 2. More complicated motion patterns are shown in coupled pendulums. These can have stable modes, such as the movement of bipeds, quadrupeds, and insects; the limbs are the pendulums; in general, they move in pairs or triples, thus simplifying the dynamics to two main forces (see Haken 1996: chap. 9, Animal gaits and their transitions). 3. Chaotic modes occur with double or multiple pendulums; the motion of the second pendulum depends on the motion of the first one. Coupled dynamics occur in films in which a series of processes is shown or where a plurality of forces interacts. This feature is central in many action films. The following examples illustrate the three types of dynamics: 1. Domino effects: one event triggers the next. Thus, a series of murders occurs in the crime film, either the murderer continues the given type (the case of Jack the Ripper) or to escape or hide his first murder he/she commits further ones. Repeated scenes of attack and flight are shown in war movies; the classical example is Eisenstein’s “Massacre on the stairs in Odessa” in Battleship Potemkin (Eisenstein 1925). 2. Chase and escape sequences in many action films and Western films (see the examples below). The underlying momentum of the persecution is: A pursues B (in space with obstacles). 3. Parallel sequences of events that are linked selectively. The events are originally independent but come into contact/interaction by chance. I will discuss this feature in the context of Bond movies later.

3.4 The Moving Camera Another important feature concerns the focus and movement of the camera: change of shot, zoom or a moving camera on a camera sled, vehicle or crane. It follows the movement of the actors and simulates the motion of an imaginary viewer. Since the 1950s, hand-held cameras are used to simulate

Movie Physics or Dynamic Patterns as the Skeleton of Movies

79

the natural movement of the viewer’s attention (the ‘subjective view’; early forms were the ‘keyhole effect’ or mask images). The Steadicam perfected the technique of hand-held cameras in the 1970s. It can be carried by a person who follows the actors or precedes them (cf. Schemikau 2006). Examples of intensive camera movement are Breathless (Godard 1960; see Figure 4.6a), Rocky (Avildsen 1976), or The Marathon Man (Schlesinger 1976; see Figure 4.6b).

Figure 4.6 (a) Belmondo in Breathless (1960) and Dustin Hoffman in The Marathon Man (1976). The camera moves in front of the actors in the street.

4 The (Re-)Construction of a Story in Movies The focus of the camera and the construction of the sequence of scenes in the editing room constitute the central levels of organization of cinematic meanings. They supplement and complete the work of the actors in their performance in front of the camera. This is a major difference from the theater where, although spatial positions, movements, and perspectives may be prepared, the result cannot be designed as detailed and radically as in subsequent construction by montage. We can thus distinguish three sub-levels of meaning-construction in film: 1. The construction of meaning on the set, in front of the camera (prepared by the script, planned by stage management, and made concrete by the actors). 2. The construction of meaning by camera work due to the choice of setting, control of lighting effects, and by the movement of the camera in space and zooming. In most cases, a multiple of the required film material is recorded, i.e., the camera creates a potential narrative space from which radical selection starts. Complementary to the captured pictures, the off or hors-champ, is used, which can contribute connotative meaning. 3. Montage. In its first stage, this is primarily privative, i.e. large parts of the film material are discarded. The film director in the editing room resembles the sculptor who shapes the figure, existing only in his/her imagination, from the marble block. The syntactic and narrative order, which is generated by montage, is basic and elaborates the rough design

80  Wolfgang Wildgen of the plot on the storyboard. In contemporary films, some scenes are computationally produced or supplemented by special effects. In some cases, even major parts of the film may be digitally produced and augmented by scenes technically made with the use of a camera (movements of real actors may be used for the animation of artificial characters that were formed on the computer).19 These three levels of meaning are essentially visual. Textual-linguistic and musical-acoustic dimensions are added. Different versions of a film may even use dialogues in other languages or be shown without music, which can be performed live by an orchestra. This demonstrates the relative independence of the three basic constituents: Image – Text – Music. The level of visual organization is structured, as shown above, organization in front of the camera (director and actor) – camera and lighting performance – cutting, montage, and special effects. The film must integrate these three levels and avoid redundancy and contradiction. The integration is done in specific zones of each organizational level, so that, in general, these remain relatively autonomous. Thus, the montage and organization of texts must match. Every change in the montage automatically changes the text structure and narrative content. Moreover, the focus of the camera must respect the weight of every person or harmonize their roles in the text. If the main person is in no way highlighted (optically via its size or its motion when the camera follows it), specific narrative threads may not be appropriately represented. The integration of music has to be coordinated with the editing and montage, but it is also tied to the narrative structure to the extent that passages of complications and the climax phase of the story should also correlate with the music. The dominant dimension is certainly the visual construction that is performed by the actors, the camera, and by cutting and montage. Overall, a sufficiently large space for expression must be provided, without which the filmic semiosis would fail for reasons of complexity. The separation of the levels of organization and their degree of autonomy is the key to the necessary reduction of complexity. In this regard, the silent movie has been a historical experiment as it has shown the degree of autonomy of the visual aspect of film (and its limits; see, for example, the analysis of Lang’s silent movie Die Niebelungen: Siegfried (1924) in Wildgen 2013b). The techniques for the mastery of complexity are roughly the same as in language. First, a standardized lexicon of pictorial and montage techniques is developed. Secondly, rules are provided for syntactical/structural organization on several levels. In this sense, we may speak of a language (grammar) of film. In the following section, I will analyze some aspects of the most recent Bond films (2006–2015), focusing mainly on the second film, Quantum of Solace (Forster 2008), in relation to action sequences, parallel actions, and acceleration or deceleration (see, for more details, Wildgen 2013a (in German) and 2014 (in English)).

Movie Physics or Dynamic Patterns as the Skeleton of Movies  81

5  Movie Physics in Action Movies (with a Focus on Bond Movies) The basic dynamics of Bond movies are already present in Hitchcock’s classic thrillers with one central male protagonist who has to fight a whole bulk of adversaries (among them also police or police authorities). This is clearly also the case in three Hitchcock films: 1 In the 39 steps (Hitchcock 1935), the hero Richard Hanney must escape the police, who think he is the murderer, and an organization of spies at the same time. In this enterprise, several young and pretty women trust and help him. 2 In The Foreign Correspondent (1940), we witness again an amateur who is drawn into complicated and dangerous adventures. 3 In North by Northwest (1959), a similar paradigm is used: The protagonist cannot imagine why he is followed and risks being killed. Central scenes of persecution and flight occur when he is waiting in a very open and lonely place and is suddenly attacked by a plane such that he has to hide in a cornfield. The recurring motive amounts to a protagonist who struggles with a large number of better equipped and organized adversaries (police and criminal gangs), although he is not involved in the underlying conflict or intrigue. In the case of the Bond films, the situation is much simpler: it is the job of the man with the license to kill to accept all the chases, struggles, and fights which occur. Furthermore, well-known places are characteristic of these films. They are usually distributed over several continents, but England remains the central point of Bond’s ‘empire’. This feature is particularly highlighted in Skyfall (Mendes 2012), which commemorates the 50th anniversary of the series. Transitory locations also play an important role, e.g. the lobbies of hotels, elevators, railway stations, airports, and crowded places, such as the market square of Siena where the finish of the traditional horse race takes place (in Quantum of Solace; Forster 2008) and the Turkish bazaar in Istanbul (Skyfall). Bond is constantly in transition, and he gets acclimated to every place he is in as if it were home. The decor of the spaces has changed over the years. In Quantum of Solace, recognized and attractive places, such as Lake Garda, Siena, London, Port au Prince, and the Bolivian highland, are preferred. In Skyfall, even the pursuit over the rooftops of Siena in the previous film is quoted in the opening scene when Bond is chasing his opponent over the rooftops of the bazaar in Istanbul (now on motorbikes instead of on foot). Such resumptions are characteristic of the movie series. The climax of action (with the consolation scene announced in the title of the film) in Quantum of Solace occurs in the Desert Hotel Perla de las Dunas (Bolivia). The successive explosion of parts of the building (right to left) is a consequence of

82  Wolfgang Wildgen a new technology employing hydrogen fuel cells. Thus, the central themes, water – water resources and water scarcity – hydrogen, come to a head.20 In Skyfall, the final confrontation occurs in a Scottish castle, the site of Bond’s childhood. Again, this place is destroyed. Besides the action scenes, destructions and explosions are a basic dynamic schema of Bond films.21

5.1  Pursuit and Acceleration in Bond Films As Kracauer remarks in his classic analysis, chases, pursuits of antagonists, are the most characteristic feature of the film medium (cf. Kracauer 1964:  72) and they were already central in early films around 1900 in France (La Course des sergeants de ville 1906) and characteristic for the climax in the movies of Griffith (cf. Kracauer 1964). They contribute to the “dynamization of space”, as Erwin Panofsky tells us (cf. Kappelhoff 2008: 24). Steven Spielberg, who later became a master of blockbuster films, produced his first thriller, The Duel (1971 on TV, 1973 in the cinema), which is practically one chase and flight on the road between a car driver and a truck driver. The motive of manhunt or tracking is central for Western films and thrillers in general. It realizes the archaic schema of chase and flight and demonstrates the motor control of the protagonists. In the following, I will investigate the film Quantum of Solace in respect to the structure and complexity of selected chase scenes. Similar structures can be detected in all Bond films and the ‘action movie’ and ‘thriller’ genres. I first describe some dynamic schemata underlying simple event and action structures (see Wildgen 1994: chap. 3 for a list of process scenarios). If we take the action and pursuit scenes in the Bond movies, we notice that in the persecution the agent first walks through the streets of a city. As in a dialogue, the persecutor (or the shadowing detective) and the persecuted are shown alternatively. The perspective and the actors change regularly at a certain rhythm. By regular repetition of the change, continuity is suggested. The time gaps in the parallel montage (crosscutting) must be filled in by the viewer. The rhythm of switching between cuts that show the persecuted and the persecutor is a basic speed which can be accelerated or slowed down. The persecutor begins to walk faster and eventually runs, and the mode of locomotion can be changed with the use of a faster means of transportation such as by walking > use of a bike > by car > or by helicopter. Characteristically, the means of transportation of the person persecuted change in correspondence: the persecuted flees on a motorcycle, the tracker uses a car, the persecuted changes also to a car, the persecutors use a helicopter, etc. In Western movies, the different agents (or groups of agents) may be on horse, or one party moves in a stagecoach or a train. This results in endless combinations whose base is always the relative speed of the agents and the way they react to obstacles on their way. In the Bond film, daredevil races are shown where the hero often fights against several and sometimes

Movie Physics or Dynamic Patterns as the Skeleton of Movies  83 better-equipped opponents and, eventually, makes use of some carefully prepared secret weapon (introduced in the first part of the film). In addition to movement and rhythm of the pursuers and the pursued, a third force can be introduced: the environment in which the fight takes place. It begins to move and change. How do the agents react to collapsing buildings, exploding rooms or the spreading fire? Containers with hot or corrosive liquids begin to leak (due to shooting of the agents) and eventually explode, etc. This type of propagation of the moving components generates a specific form of perceived acceleration. Of course, this progression must find a result. Thus, the pursuer may lose track of the pursued, the persecuted becomes a victim of the collapsing rooms or finally succumbs in confrontation with the protagonist. In the classical Western movie, action almost comes to a standstill before the climax (the duel of the rivals). The final shootout finally resolves the tension. In Quantum of Solace, the viewer is granted just short breaks. Only at the end, when everything is decided, is the film decelerated. Action movies often have a standard target: the protagonist, who stands for positive values, wins after endless disputes. In the Bond movie, this game is not satirically or ironically reflected (as in some late Western films) but is taken less seriously (it is just for the fun of the audience, which is not frightened).

5.2 The Dynamics of more Complex Chases in Bond Movies Quantum of Solace already shows a car chase after the opening credits: cars hunting each other along Lake Garda and the quarries of Carrara in Italy. The persecutor must first reduce the advance of the persecuted until the latter comes into view or in firing range (this phase is skipped in the opening episode of the film, it may be represented with great detail in a Western film). This leads to the proper action phase. A simple shootout does not yet decide the chase (this comes along with making everything riskier). The cars eventually come into contact, repel, push one another to the side, cut their way, etc. It may even come to eye contact between the drivers (if they drive in parallel). In Quantum of Solace, Bond shoots into the car beside him, i.e. he has eye contact with the driver. Characteristically, a third party comes into play: traffic in the opposite direction (often trucks, tank cars or construction trailers, i.e. large, heavy obstacles) or the police who tries to intervene (usually in vain). Indirectly, the road can turn out as a kind of antagonist: tight curves and dangerous knolls, steep roadsides or dark tunnels, roadwork and traffic congestion must be taken into account by the drivers (see the screenshots in Figure 4.7). The basic scenario is one of catching and eliminating. This is graphically abstracted in the schema in Figure 4.7b related to the archetype of capture in catastrophe theory (cf. Wildgen 1982: 42–43).

84  Wolfgang Wildgen

Figure 4.7  (a) Screenshots from Quantum of Solace: place of chase are a road tunnel at Lake Garda and the marble quarries of Carrara; (b) a schematic description of the chase and the archetype of capture below (cf. Wildgen 1982).

The first scene of the film escalates the chase theme. It takes place on two levels. On the first level, Siena is the place where traditional horse races are held in the city center. These take place simultaneously with the meeting of Bond. On the second level, the prisoner can escape, and Bond pursues him across the rooftops of Siena. Bond’s hunt goes above the rooftops, finally through a flat and to a bell tower. From the bell tower, both opponents fall onto an unstable scaffold with a rotating freight elevator, and the elevator cables are used for a fight in the ‘circus dome’. At this point, the linear model is left, and the pendulum-like movements take place in three-dimensional space, i.e. not along a path on a surface. The singularities, i.e. the points of local contact and conflict of both actors, are complex in a geometrical sense and difficult to control. In addition, both lose their weapons which are swept away by the movement of the scaffold and the ropes. Both must get hold of their weapons for the final shootout.

Figure 4.8  Coupled pendulums (the two ropes) and double pendulums (the arm on which the rope hangs also moves).

Movie Physics or Dynamic Patterns as the Skeleton of Movies  85 A kinematic complication is added to the spatial one. The movement of Bond is dependent on the oscillation of the rope and the rotation of the boom to which the rope is attached. Bond, at the last moment, catches his weapon and shoots the opponent, who has also found his gun. The extended model can be described as a coupled pendulum (if both agents oscillating with the ropes touch each other) or as a double pendulum (if the boom, on which the rope hangs, moves). From physics (and chaos theory), we know that the double pendulum has chaotic phases, i.e. its motions cannot be strictly controlled. Coupled pendulums may have harmonic motion modes, but these are accessible only under special conditions. The viewer is obviously not a chaos theorist, but can understand the increasing uncontrollability (almost physically) and thus estimate the extreme skill of the protagonist Bond.22 At the same time, this design points to a fundamental human need that shows up in gambling. The players live in the illusion that they can control a complex situation, which, in fact, they cannot. Sutton-Smith (2007: 173) speaks of an illusory “ego-mastery”. He interprets this behavior as an expectation of divine help (Sutton-Smith 2007: 157; the reference to Homer’s Odyssey is obvious). The run up to the chase scenes is followed by a short resting phase with the emptying square in Siena, where onlookers are leaving the area. This is also a repeating pattern. Thus, after a chase by boat (also a classic motif in Bond films), Bond directs the boat into a sunny harbor where tourists are embarking. He hands over the unconscious Camille to one of the tourists. After the climax, the story usually ends with a coda, in which Bond and the Bond girl are peacefully united. For example, at the end of the first James Bond film Dr. No (Young 1962), Bond hugs the Bond girl while sitting in the jolly boat. In Quantum of Solace, the fight scenes in the hotel end with a kind of reconciliation among the surviving protagonists, Bond and Camille. But their journey will separate later. In the Jubilee film of 2012, Skyfall, Bond keeps his dying boss M in his arms at the end of the big fight. In Spectre (2015), the classical happy end is readopted. Bond leaves London with the Bond girl in his refurbished Aston Martin.

5.3 Parallel Actions and Points of Contact/Coordination The pattern of only selectively connected but parallel action sequences, which was characteristic of the episode in Siena, is repeated in Bregenz (Austria). While the opera Tosca – incidentally an early melodrama of secret service – approaches its point of culmination, the murder with a knife, Bond disturbs the secret meeting of Greene and his partners. The pursuit goes through the foyer, into the kitchen, and then ends in a duel with another agent. A dramatic chase in Bolivia then has the following constellation: an old aircraft (with Bond at the control column) and two pursuit planes. Bond and Camille finally save their lives by parachute, whereas the interceptor planes crash on a mountain. Overall, even if these actions take place at

86  Wolfgang Wildgen different locations, the narrative is held together by these boiling hunts, which are the dramatic core of the ritualized story. The last and decisive fight scene brings a new combination into play: Camille wants to kill the murderer of her family, a Bolivian general, who is currently in the hotel Las Dunas, air-conditioned by hydrogen fuel cells. Bond wants to get hold of Greene, who negotiates with the general. Both fights are simultaneously taking place after Greene has left his meeting with the general. At the first shooting by Bond, the fuel cells begin to explode.23 Thus, three processes run in parallel since the explosion of the hotel is a third force, not controlled by either party. The film presents the two fights in alternating shots: Camille – General and Bond – Greene (see Figure 4.9, second row). They are coupled when Bond hears a shot. Greene says: “You have again lost someone”. In the following scene, Bond leaves Greene to find Camille.

Figure 4.9  Screenshots from Quantum of Solace. First row: Fight in the theater foyer (exchange of gunfire in the movie); fight on stage in Bregenz (in the opera Tosca). Tosca stabs Scrapia, the blackmailer. Second row: Parallel fighting in the exploding hotel: Camille against general – Bond against Greene.

5.4 Probability Estimation and the Dynamics of Suspense A central feature of action and crime films concerns suspense. They share this feature with novels, i.e. suspense is not just a visual effect. It rather concerns the information dynamics inherent in written stories and filmed ones. Wuss (1996: 52) mentions three conditions for tension: 1. the probable occurrence of a relevant (often menacing) event in an undefined course of events, 2. the possibility of the protagonists being able to be active in bringing the events under control by certain forms of conduct (i.e. preventing the negative outcome of events),

Movie Physics or Dynamic Patterns as the Skeleton of Movies

87

3. there is a difference between the information viewers have about the uncertain situation and the kind of information to which the protagonists are privy. The reader/viewer can guess the probability of different issues or types of denouement. He/she may have a natural sympathy with one party, e.g. the innocent hero in Hitchcock’s North by Northwest, who tries to escape his hunters. If the persecutors are superior (on a plane or armed, in greater number and anonymous or disguised), the spectator will guess a very low probability of survival/success of the hero. This creates a tension between their wish/sympathy and the most likely outcome. If this situation of tension is iterated, i.e. the hero escapes in spite of very low probabilities again and again; an increasing vector of suspense is created which must finally be resumed in a climax, in the ideal case with the survival or even victory of the hero. In the case of love stories, emotional risks may accumulate and be finally dissolved. In many films, suspense distracters are introduced, so-called MacGuffins, where some of the guesswork of the audience is attracted by an element which finally shows up to be irrelevant for the denouement. In films, it is important to keep the level of suspense over long periods (with variations of intensity). It should only drop at the end of the story.

5.5 Mastery of Space and the Art of Dynamic Balance Since acceleration quickly reaches its limits, the action film must connect phases of acceleration and deceleration.24 This results in a basic rhythm of fast versus slow, to which the narrative must be adjusted and which also controls background music. Phases of partial completion (sub-gestalts) divide the film like chapters in a book. Overall, a balance of intensive action/acceleration, deceleration/ calmness is sought. As there are many ways to achieve such a balance, it becomes a characteristic for different film genres. The balance of slow versus fast has a counterpart in the balance of affective values (love versus hatred). The emotional dynamics define specific expectation horizons, which in turn generate tension. Rhythms of change may then distinguish individual film genres and films (see, for this topic, Eidsvik 2007). The mass media of film and television use general human perception and understanding skills, but the learning process, especially with an audience experienced with films, is also relevant. Therefore, one should distinguish between general laws and cultural codes. The latter are historically variable. Many contingent factors are relevant, e.g. political backgrounds, theories, ideologies, individual motivations of the film-makers or preferences of a specific audience. Different styles are created which appear and disappear almost like clothing fashions or styles in architecture or literature. The Italian ‘Neorealismo’ (after World War II, Rosselini, de Sica, Visconti, Fellini) and the French ‘Nouvelle Vague’ (late 1950s and 1960s; Godard, Truffaut, Chabrol)

88 Wolfgang Wildgen avoided the linearity and smoothness of action in Hollywood films, which Hitchcock had brought to perfection. The filmic narrative presents sequences of fragmentary and dismembered encounters (cf. Aab 2014: 180) and, rather than offering a coherent image of actions, a mental image of situations and social relations is created.

6 Conclusion The analysis of this chapter has shown that typical signs in cinema are kinematic and dynamic in their nature, i.e. they refer to motion, locomotion, action (chase, flight, fight). If action and conflict are not the central concern, emotional events, dialogues, and changes in the social, political status or of emotional constellations offer the relevant dynamics (emotion instead of motion). A frequent consequence of conflict is the disappearance or death of agents, the breaking of alliances and love connections, and the building or consolidation of new ones. In his novel ‘Wahlverwandtschaften’ (1809; filmic adaptation in Elective Affinities, Taviani/Taviani 1996), Goethe used chemical metaphors as a background for emotional transformations, i.e. he applied chemical/physical laws in literature and thus prefigured a kind of literature physics. Here, a focus on motion, forces, and the control of motion and stable states gives a new relevance to space and its organization in one, two, or three dimensions. The linguistic bias with its preference for concatenation (one-dimensionality and discreteness) is then overthrown, and the door for a proper treatment of non-linguistic (visual, musical, olfactory, etc.) signs in our real world is opened. The tools of dynamic systems theory (attractors, vectors, bifurcations, catastrophes, chaos) may be used to specify further the inventory of types which reoccur in many semiotic (and non-semiotic) systems. They can complete (if not replace) the purely relational, if not static, terms of logics which have dominated semiotics in Hjelmslev’s structuralism (and later in the semiotics of Eco and Greimas) and in linguistic schools inspired by Analytic Philosophy (e.g. in the Chomskyan paradigm). The dialogue with natural sciences relevant in this field (physics, chemistry, biology, cognitive sciences) thus requires new conceptualizations and a proper way of applying mathematics.

Notes 1. See Liu (2007) for a computational analysis of attention in movie audiences. 2. In the history of modern music, one can find a similar constellation at the beginning of the 20th century. When Arnold Schönberg (1874–1951) and others introduced the technique of 12-tone scales and seriality, their repertoire of musical forms was first small and only short pieces were possible. With the help of parallel texts, songs and dramatic (linguistic) plots, larger compositions became accessible. Thus narrativity rooted in language routines was used as a means

Movie Physics or Dynamic Patterns as the Skeleton of Movies

3.

4.

5. 6.

7. 8. 9.

10.

11. 12. 13. 14.

89

to create compositions which were able to hold the attention of a public for a longer period. In auditory perception this would correspond to the “extended present”, where perceptual units at the lower level (up to 3 seconds) are sequentially ordered. It is assumed to reach its maximum after 10 to 30 seconds (cf. Kühl 2007: 104, 107–108). In theatre, pauses between acts subdivide the story into two, three, or four pieces and the audience can reflect on the play, talk about it, etc., before the play is resumed. This is not done in films and the producer must try to sustain the attention of the audience with the means of surprise and suspense. The long scene is a classical MacGuffin (see below for further explanation), because there is no visible link to the overall story besides the fact that the hero is in danger. Hitchcock had a training as engineer before he started his movie career. In the interview with Truffaut he said: “I had acquired some practical knowledge of force and motion” (Truffaut 1968: 23). The term “physics” was used in the context of comics (“physics of comics”) for a genre which does not have the technical realism of photography (see for the field portrait, photography, comics, and movie Wildgen 2013a: chap. 7) Some aspects described in the following sections have been treated in Wildgen (2014) and further developed in Wildgen (2015). Einstein showed that acceleration is not infinitely variable; it is limited by the speed of light. In the neighborhood of a speed near the speed of light, the Galileo transformation has to be exchanged by the Lorentz transformation. Thom (1990/1988) coined the term semiophysics (“sémiophysique”). Petitot (1992) used the term physics of meaning (“physique du sens”) and established a link to the thought of Kant in his philosophy of nature and his critique of reasoning and to Husserl in his phenomenology. Classical disciplines which are bridges between natural and human sciences are: psychophysics, initiated by Gustav Theodor Fechner in 1860 that influenced the semiotician Charles Sanders Peirce. Modern psychophysics applies quantitative and experimental methods in the analysis of perception and cognition. Another very successful bridge-discipline is biophysics, which received a strong impulse with the chemical and molecular analysis of DNA by James Watson and Francis Crick (Nobel prize for medicine in 1962). Sociophysics was initiated by Weidlich and Haag (1983), who applied the methods of synergetics (cf. Haken, 1983) and statistical dynamics to social systems, e.g. to processes of migration. For an application in the sociolinguistic context, see Wildgen (1986). The areas of neuropsychology and physics are bridged in the work of Scott Kelso (e.g., Kelso 1995). See Wildgen (2004) for the evolution of language in the context of bodily evolution and physical contexts. In an interview, Alfred Hitchcock says: “Well, for one thing, the chase seems to me the final expression of the motion picture medium” (Gottlieb 1995: 125). See Saint-Martin (1990: 208) for the reading of pictures and the network of links between spatially separated visual fields. In Young and Innocent (Hitchcock 1937), the camera focuses in a fast move from the highest position, next to the ceiling to a close-up of the drummer and his eye. The twitch of his eye unveils his identity (as murderer) to the audience.

90 Wolfgang Wildgen

15.

16.

17.

18.

19.

20. 21. 22. 23. 24.

In contrast, the protagonists have to find another way (helped by chance) to identify the murderer (cf. Truffaut 1968: 93). A classical precursor is the discussion in the 18th century in European Enlightenment about the relation between beauty and truth. Untrue stories or pictures cannot be beautiful. The truthfulness of communication is even a problem in animal communication (cf. Honavar/Ma 2015). Actually, many theatre directors avoid realistic outfits on stage or in costumes of actors, and thus a parallelism to movies. Instead of “disguised representation” (cf. Dromm, 2008: 194–195), film-makers can call the attention of the public to the status of film as representation. This is the case in the film Dogville (2003). Probably all artwork moves on a scale between perceived (imagined, dreamed) reality and controlled illusion, if not fake. See Orson Welles’ comments in his film F for Fake, his last major film released in 1974. In cartoons the restrictions concerning realism seem to be less strict. Nevertheless, Bukatman (2014) tells us in reference to the cartoons of Tom and Jerry produced by Goldwyn-Mayer between 1940 and 1957: “But Hollywood cartoons do not give us an entirely disordered universe of chaos and entropy. They give us a world that is ordered differently: hence cartoon physics” (Bukatman 2014: 301–302). In 2002, a documentary film called Best ever Bond showed a selection of the ten ‘bests’ scenes in forty years of Bond films; for the list of films, see: http:// programm.ard.de/TV/daserste/best-ever-bond/eid_281066292338027?list=now [last accessed: 1 March 2016]. Many of them are action and fight scenes. In Steven Spielberg’s film The Adventures of Tintin. The Secret of the Unicorn (Spielberg 2011) the Performance-Capture-Technology is used to create a “skeleton” of the moving actor, which is then covered with the shape of the comic figures, e.g. Tintin, Captain Haddock, etc. Moreover, this technique allows an artificial acceleration of movements and thus some effects difficult to achieve with real actors (cf. Schickel 2012: 250–255). In reality the building was an astronomic observatory in Peru. The destruction was filmed using a model in the production hall. In Christian (Greek and Jewish) myths of the final destruction of the world and the ultimate judgement (apocalypse), this is a major feature. It reappears as a central motif in many action films. In fairgrounds, complicated multiply rotating carousels are special attractions. In this sense, the action film imitates such fairground attractions. The burning Hotel Perla de las Dunas corresponds to the collapsing scaffolding during the chase scene in Siena. From the perspective of the audience, human locomotion (supported by horses or machines) is the standard of comparison. The upper limit of speed in relation to the spectator or the moving camera is given by the human faculty of visual tracking. Speeds beyond this threshold are irrelevant.

Bibliography Aab, V. (2014). Kinematographische Zeitmontagen: Zur Entwicklungsgeschichte des Kinos. Marburg: Schüren. Anderson, J. D./ Fisher Anderson, B. (eds.) (2007). Narration and Spectatorship in Moving Images. Cambridge: Cambridge Scholars Publishing.

Movie Physics or Dynamic Patterns as the Skeleton of Movies

91

Beckman, K. (ed.) (2014). Animating Film Theory. Durham: Duke University Press. Biro, Y. (2008). Turbulence and Flow in Film. The Rhythmic Design. Bloomington: Indiana University Press. Bukatman, S. (2014): “Some Observations Pertaining to Cartoon Physics; or, The Cartoon Cat in the Machine”. In: Beckman, K. (ed.). Animating Film Theory. Durham: Duke University Press, chap. 17. Burch, N. (1983): Theory of Film Praxis. London: Secker. Chapman, J. (2007). Licence to Thrill. A Cultural History of the James Bond Films. Second Edition. London: I.B. Tauris. Dromm, K. (2008). “Spielberg and Cinematic Realism”. In: Kowalski, D. A. (ed.). Steven Spielberg and Philosophy. We’re Gonna Need a Bigger Book. Lexington: The University Press of Kentucky, pp. 191–209. Eidsvik, C. (2007). “Neurotransmitter Persistence, Narrative Structure, and Emotion: A Realistic View”. In: Anderson, J.D./Fisher Anderson, B. (eds.). Narration and Spectatorship in Moving Images. Cambridge: Cambridge Scholars Publishing, pp. 42–48. Gottlieb, S. (1995). Hitchcock on Hitchcock. Selected Writings and Interviews. London, faber and faber. Haberlik, C./Busch, A. C. (2003). 50 Klassiker. Architektur des 20. Jahrhunderts. Hildesheim: Gerstenberg. Haken, H. (1983). Synergetics. An Introduction. Third revised and augmented edition. Berlin: Springer. Haken, H. (1996). Principles of Brain Functioning. A Synergetic Approach to Brain Activity, Behavior and Cognition. Berlin: Springer. Honavar, V./Ma, Z. (2015). “Towards Computational Models of Animal Communications, an Introduction for Computer Scientists”. Cognitive Systems Research 33, pp. 70–99. Kappelhoff, H. (2008). Realismus. Das Kino und die Politik des Ästhetischen, Berlin: Vorwerk 8. Kelso, J.A.S. (1995). Dynamic Patterns. The Self-Organization of Brain and Behavior. Cambridge: MIT Press. Kracauer, S. (1984). Theorie des Films. Die Errettung der äußeren Wirklichkeit. Frankfurt a.M.: Suhrkamp. Kramer, M. (2007). “The Epistemology of Gaze”. In: Anderson, J. D./ Fisher Anderson, B. (eds.). Narration and Spectatorship in Moving Images. Cambridge: Cambridge Scholars Publishing, pp. 136–147. Kühl, O. (2007). Musical Semantics. Bern: Peter Lang. Liu, Anan/Li, Jintao/Zhang, Yongdong/Tang, Sheng/Song, Yan/Yang, Zhaoyuan (2007): “Human Attention Model for Action Movie Analysis”. IEEExplore. Digital Library. 1–4244-0971-3/07. Metz, C. (1968). Essais sur la signification au cinéma. Paris: Klincksieck. Petitot, J. (1992). Physique du sens. De la théorie des singularités aux structures sémio-narratives. Paris: Éditions du CNRS. Petitot, J. (2003). Morphogenesis of Meaning. Bern: Peter Lang. Prange, R. (2012). “Zur Theoriegeschichte der filmischen Raumkonstruktion und ihrer Aktualität als Gegenstand einer historischen Bild- und Medienwissenschaft”. In: Engelke, H./Fischer, R.M./Prange, R. (eds.). Film als Raumkunst. Historische Perspektiven und aktuelle Methoden. Marburg: Schüren, pp. 12–53. Saint-Martin, F. (1990). Semiotics of Visual Language. Indiana: Indiana University Press.

92 Wolfgang Wildgen Schemikau, M. (2006). “In Bewegung – der schwebende Blick der Steadycam”. In: Koebner, T./Meder, T./Liptay, F. (eds.). Bildtheorien und Film. München: edition Text + Kritik, pp. 316–334. Schickel, R. (2012). Spielberg. Seine Filme, sein Leben. München: von dem Knessebeck. Souriau, É. (1969). La correspondance des arts. Paris: Flammarion. Spoto, D. (1976). The Art of Alfred Hitchcock. Fifty Years of his Motion Pictures. New York: Dolphin Book. Sutton-Smith, B. (2007). “The Ambiguity of Play. Rhetorics of Fate”. In: Bial, H. (ed.). The Performance Study Reader. Second edition. London: Routledge, pp. 152–158. Thom, R. (1990). Semio physics. A Sketch. Addison-Wesley, Redwood City, Calif. (French original: Esquisse d’une sémiophysique: physique aristotélienne et théorie des catastrophes, 1988. Paris: Interéditions). Tolan, M./Stolze, J. (2008). Geschüttelt und nicht gerührt. James Bond und die Physik. München: Piper. Truffaut, F. (1968). Hitchcock. London: Secker & Warburg. Vorderer, P./Wulff, H.-J./Friedrichsen, M. (eds.) (1996): Suspense. Conceptualizations, Theoretical Analyses, and Empirical Explorations. Mahwah: Lawrence Erlbaum. Watzlawick, P./Jackson, D.D./Bavelas, J.B. (1967). Pragmatics of Human Communication. A Study of Interactional Patterns, Pathologies, and Paradoxes. New York: Norton. Weidlich, W./Haag, G. (1983). Concepts and Models of a Quantitative Sociology. The Dynamics of Interacting Populations, Berlin: Springer. Wildgen, W. (1982). Catastrophe Theoretical Semantics. An Elaboration and Application of René Thom’s Theory. Amsterdam: John Benjamins. Wildgen, W. (1986). “Synergetische Modelle in der Soziolinguistik. Zur Dynamik des Sprachwechsels Niederdeutsch-Hochdeutsch in Bremen um die Jahrhundertwende (1880–1920)”. Zeitschrift für Sprachwissenschaft 5(1), pp. 105–137. Wildgen, W. (1994). Process, Image, and Meaning. A Realistic Model of the Meanings of Sentences and Narrative Texts. Amsterdam: John Benjamins. Wildgen, W. (2004). The Evolution of Human Language. Scenarios, Principles, and Cultural Dynamics. Amsterdam: John Benjamins. Wildgen, W. (2013a). “Erzählung und Action im James Bond-Film: Ein Quantum Trost”. In: Bateman, J.A./Kepser, M./Kuhn, M. (eds.). Film_Text_Kultur. Beiträge zur Textualität des Films. Marburg: Schüren, pp. 321–345. Wildgen, W. (2013b). Visuelle Semiotik. Die Entfaltung des Sichtbaren. Vom Höhlenbild zur modernen Stadt. Bielfeld: transcript. Wildgen, W. (2014). “Dynamic Meaning Construction in Action-films (specifically in the last James Bond Trilogy)”. RASK: Internalionalt Tidsscrift for Sprog og Kommunikation 40, pp. 423–448. Wildgen, W. (2015). “Catastrophe Theory and Semiophysics: with an Application to ‘Movie physics’”. Language and Semiotic Studies 1(2), pp. 61–88. Wildgen, W./Brandt, P.A. (eds.) (2010). Semiosis and Catastrophes. René Thom’s Semiotic Heritage. Bern: Peter Lang. Wuss, P. (1996). “Narrative Tension in Antonioni”. In: Vorderer, P./Wulff, H.-J./ Friedrichsen, M. (eds.) (1996): Suspense. Conceptualizations, Theoretical Analyses, and Empirical Explorations. Mahwah: Lawrence Erlbaum, pp. 51–70.

Movie Physics or Dynamic Patterns as the Skeleton of Movies  93

Filmography 39 Steps, The (1935). Alfred Hitchcock. Gaumont-British Picture Corporation. USA. Adventures of Tintin, The. The Secret of the Unicorn (2011). Steven Spielberg. Paramount Pictures/Columbia Pictures. USA/New Zealand. Battle Ship Potemkin (1925). Sergei Eisenstein. Goskino. USSR. Bonny and Clyde (1967). Arthur Penn. Warner Brothers. USA. Breathless (1960). Jean-Luc Godard. UGC. France. Casino Royale (2006). Martin Campbell. Columbia Pictures. USA/UK. Die Nibelungen: Siegfried (1924). Fritz Lang. Ufa. Germany. Dr. No (1962). Terence Young. Eon Productions. USA. Dogville (2003). Lars von Trier. Lions Gate Entertainment. Denmark. Duel (1971). Steven Spielberg. Universal Studios. USA. Elective Affinities, The (1996). Paolo and Vittorio Taviani. Filmtre. Italy/France. F for Fake (1975). Orson Wells. Speciality Films. France. Family Plot (1976). Alfred Hitchcock. Universal Pictures. USA. Foreign Correspondent (1940). Alfred Hitchcock. United Artists. USA. Goldfinger (1964). Guy Hamilton. United Artists. UK/USA. La course des sergents de ville (1907). Ferdinand Zecca. Pathé Frères. France. Marathon Man, The (1976). John Schlesinger. Paramount Pictures. USA. Metropolis (1927). Fritz Lang. Ufa. Germany. Moonraker (1971). Lewis Gilbert. United Artists. UK. North by Northwest (1959). Alfred Hitchcock. Paramount Pictures. USA. On her Majesty’s Secret Service (1964). Peter R. Hunt. United Artists. UK. Quantum of Solace (2008). Marc Forster. Metro-Goldwyn-Mayer/Columbia ­Pictures. UK/USA. Queen Christina (1933). Ruben Mamoulian. Metro-Goldwyn-Mayer. USA. Rocky (1976). John G. Avildsen. United Artists. USA. Rope (1948). Alfred Hitchcock. Transatlantic Pictures. USA. Searchers, The (1956). John Ford. Warner Brothers. USA. Skyfall (2012). Sam Mendes. Metro-Goldwyn-Mayer/Sony Pictures. USA/UK. Spectre (2015). Sam Mendes. Metro-Goldwyn-Mayer Pictures/ Columbia Pictures. USA/UK. Spy who Loved me, The (1977). Lewis Gilbert. United Artists. UK. Truman Show, The (1998). Peter Weir. Paramount Pictures. USA. Vertigo (1958). Alfred Hitchcock. Paramount Pictures. USA. Young and Innocent (1937). Alfred Hitchcock. GFD Gaumont Film Company. UK.

5

From Visual Narrative Grammar to Filmic Narrative Grammar The Narrative Structure of Static and Moving Images Neil Cohn

1 Introduction The question of how we understand sequences of communicative images applies equally to static sequential images – as in the visual narratives found in comics – and to moving sequential images – as in the visual narratives found in film. Many assume that similar underlying principles of comprehension apply across both static and moving images, though they present information in ways that can depart radically. This paper investigates the comprehension of sequential images by exploring how the theory of Visual Narrative Grammar (VNG) (Cohn 2013b) overlaps with notions from film theories and can apply to films. Since VNG has been designed as a theory for static sequential images (like those found in comics), we will, in particular, explore how moving images change this structure to suit the affordances of the medium, adapting to become a Filmic Narrative Grammar (FNG).

2 The Challenge at Hand Drawn sequential visual narratives have appeared throughout human history (Kunzle 1973; McCloud 1993), yet until the advent of film, the illusion of motion had been fairly rudimentary and limited in scope. Despite this longevity for drawn narratives theories about the structure of films have been far more prevalent and have a longer contiguous history than most theories of static visual narrative comprehension. In both cases, early theories appealed to unitary inferential processes, like montage (Eisenstein 1942) or closure (McCloud 1993) for almost all relations between images. More recent theories from psychology have stressed that film comprehension, and the development of its editing, adapts the preferences of general cognitive processes related to attention, perception, and event segmentation (Levin/Simons 2000; Berliner/Cohen 2011; Smith 2012; Zacks 2014). These theories link well with psychological approaches to discourse and narrative which stress the monitoring of linear semantic relationships throughout a visual narrative (Zwaan/Radansky 1998; Magliano/Zacks 2011; Zacks 2014), and echo the classic focus on the inferential processes fusing together images or film shots (Eisenstein 1942; McCloud 1993). Such

From Visual Narrative Grammar to Filmic Narrative Grammar

95

models are also consistent with theoretical approaches to visual narratives that emphasize the dynamic updating of a mental model with semantic information (e.g., Wildfeuer 2014; Bateman/Wildfeuer 2014). Interestingly, these same basic constructs are posited as not only guiding our understanding of visual narratives, but also our understanding of daily life (Radvansky/Zacks 2014; Zacks 2014). Given that all individuals should share basic perceptuo-semantic cognitive processes, such understandings should, therefore, be universal and uniform. However, why then are some individuals who lack exposure to film unable to understand certain sequences (Schwan/Ildirar 2010; Ildirar/Schwan 2015)? And, why are some individuals unable to comprehend or produce coherent drawn sequential images without experience with graphic narratives (e.g., Fussell/Haaland 1978; Byram/Garforth 1980; Wilson 2016), even with exposure to film (Wilson/Wilson 1987)? Rather, exposure and practice with a graphic system matter, evident even in neural responses to narrative patterns, which are modulated by readership of specific types of visual narratives that use those patterns (Cohn/Kutas in preparation). In addition, while inference and general perception may help explain aspects of sequential image comprehension, they do little to explain production. That is, they cannot explain the processes governing how an author selects which image comes next, especially when such sequences are drawn, not ‘edited’ together. A process of dynamic updating or perceptuo-semantic inference only explains a backward-looking process integrating what has already been experienced, but does not constrain what might be produced or comprehended ahead of what has already been read or drawn. Comprehenders do make such forward-looking predictions in visual narratives (Cohn/Paczynski 2013; Cohn et al. 2014). Furthermore, growing evidence supports that comprehenders do not rely solely on changes in meaning between images in a sequence (Cohn et  al. 2012; Cohn et al. 2014; Cohn/Bender 2016). Experimentation has long shown that images form groupings into constituents beyond linear relations (Gernsbacher 1985; Hagmann/Cohn 2016), and such groupings rely on forward-looking predictive processes, not just backward-looking reanalysis of prior relations (Cohn et al. 2014). Sequences may also have structural features that cannot be processed linearly, such as long-distance connections between images, center-embedded clauses, and structurally ambiguous sequences with several coherent parsings (Cohn 2013b, 2010). And again, such structures are not dependent on semantic relations (Cohn/Bender 2016; Cohn et al. 2014; Cohn/Kutas in preparation). Altogether, these issues warrant a theory of sequential image structure that extends beyond inference and general perceptuo-semantic processes alone, as in the emergent psychological literature on film. At the same time, theoretical accounts of visual narrative structure must offer more than just a promissory note as to the underlying cognitive processes, by providing testable predictions that are in turn supported by the psychological literature.

96

Neil Cohn

3 Visual Narrative Grammar Visual Narrative Grammar (VNG) outlines a model of sequential image understanding integrating linguistic theory and psychological experimentation (Cohn 2013a). In that it draws an analogy between language and sequential images, VNG is similar to previous ‘linguistic’ approaches to film (Metz 1974), including those that have attempted to model filmic ‘grammar’ (Carroll 1980; Chateau 1987; Colin 1995; Buckland 2000). However, unlike previous theories, VNG is based on contemporary linguistic theories that separate grammar from the meaning (Jackendoff 2002; Culicover/ Jackendoff 2005), rather than on outdated ‘structuralist’ theories (e.g., Metz 1974) or Chomskyan phrase structure grammars (Chomsky 1965). These approaches both retain ambiguous relations between structure and meaning (e.g., Carroll 1980; Colin 1995; Buckland 2000) and do not account for decades of psychological research showing such a separation (e.g., Miller/ Isard 1963; Marslen-Wilson 1987; Osterhout/Nicol 1999). In addition, unlike many theoretical models, VNG’s claims are grounded in linguistic methods like diagnostic tests (Cohn 2013b, 2014a) that form the basis for psychological experimentation (Cohn 2014b; Cohn/Wittenberg 2015; Cohn/Bender 2016). These studies have shown that similar neural responses appear to manipulations of VNG as to manipulations of syntax in sentences and that these are different from the neural responses to semantics (Cohn et al. 2012/Cohn et al. 2014). VNG is thus distinguished not only from older ‘grammatical’ models but also from models of sequential image understanding that posit only meaningful relations between units (Magliano/Zacks 2011; Bateman/Wildfeuer 2014). Such evidence grounds VNG in empirical, psychological research that extends beyond just theoretical claims. VNG argues that semantic cues within images in a sequence map to narrative roles organized within hierarchic constituents at a ‘discourse’ level of meaning, analogous to the way that syntactic categories organize words into constituents at a sentence level (Cohn 2013b). Though VNG has been designed to describe the structure of drawn sequential images (as in comics), I argue here that it can also apply to filmic narratives, which will be described in depth. VNG posits several basic narrative categories1: Establisher (E) – sets up an interaction without acting upon it, often as a passive state Initial (I) – initiates the tension of the narrative arc, prototypically a preparatory action and/or a source of a path Peak (P) – marks the height of narrative tension and point of maximal event structure, prototypically a completed action and/or goal of a path, but also often an interrupted action Release (R) – releases the tension of the interaction, prototypically the coda or aftermath of an action

From Visual Narrative Grammar to Filmic Narrative Grammar

97

These descriptions outline the prototypical correspondences of narrative categories to meaning –the ways that the semantic content (the visual cues within images) may influence the structural role that an image plays in a sequence. Nevertheless, narrative categories in VNG are identified both by a panel’s bottom-up content and its top-down context in a global sequence, as determined by distributional tendencies throughout a narrative sequence (Cohn 2013b; 2014b). This is analogous to how syntactic categories (like nouns, verbs) prototypically correspond to the semantics (like objects, events) of words (Jackendoff 1990), but also rely on sentence context. The canonical narrative schema in VNG places these categories in this order: Canonical narrative schema [Phase X (Establisher) – (Initial) – Peak – (Release)] This schema outlines that a ‘phase’ (a narrative constituent) consists of these categories in this order. Not all phases need to contain all categories, and thus the non-obligatory elements are in parentheses. Because Peaks motivate a sequence (as the ‘head’ of the phase), it is not marked as optional. However, even Peaks can be omitted felicitously under certain constrained, inference-generating conditions (Cohn/Kutas 2015; Magliano et al. 2015). Note that this narrative schema is not a ‘rule’ in the sense of traditional phrase structure grammars (e.g., Chomsky 1965 for syntax; Mandler/Johnson 1977 for narrative; Carroll 1980 for film), but rather is  a ‘construction’ stored in memory as an abstract schematic pattern, akin to syntactic patterns in the lexicon of language (Goldberg 1995; Jackendoff 2002). Narrative categories do not just apply to individual images, but each category can expand into its phase. Consider the sequence in Figure 5.1. In the first panel, a boxer reaches back to punch another, which is a preparatory action prototypical of an Initial. The punch is then completed in the next panel, a Peak in relation to that Initial. The next panel resets the actions with an Establisher in panel 3, which sets up a new situation in a relatively passive action of the boxers standing across from each other. Another Initial appears in panel 4, again with a preparatory action. The next panel ‘zooms in’ on the information in the prior Initial with a spatial modifier of a Refiner (discussed below). The penultimate panel, a Peak, does not depict a completed action, but rather shows an unexpected interruption of the boxer’s action: slipping and making the punch unrealized. The final panel shows the coda of the action in the prior Peak, here with a Release showing the victor standing over his knocked-out opponent. Beyond the roles that panels play, this sequence also embeds panels into hierarchic structures. As is, the surface pattern of I-P-E-I-Ref-P-R does not conform to the canonical narrative schema. However, its parts do, if combined to form constituents. Thus, complexity can primarily be introduced into a narrative by expanding categories into their constituents. Any

98

Neil Cohn

Figure 5.1 A narrative sequence with two narrative constituents and one subordinate modifying constituent with a Refiner.

grouping of panels can play a role in a larger structure, and when they reach a maximal node, the whole is considered an ‘Arc’. In Figure 5.1, the first two panels form an Initial constituent that together set up a Peak constituent comprised of the remaining five panels. Each of these constituents maintains the canonical narrative schema within their scope. As discussed before, Peaks form the ‘head’ of each phase, now indicated by double-barred lines to show that they motivate the primary meaning of their superordinate phase (i.e., each constituent is an expansion of its Peak). Thus, narrative categories apply recursively to individual panels and whole constituents. These principles also extend further upward – the same principles that guide short sequences govern higher ‘plot’ level narrative structures. VNG expands on the canonical narrative arc using several modifiers, such as Refiners, which repeat information found in another panel to give it a ‘refined’ viewpoint (Cohn 2013, 2015b). In Figure 5.1, the second Initial expands into a phase when modified by a Refiner of the puncher alone. Here, the larger viewpoint panel becomes the motivating ‘head’ panel (double bar lines), and the Refiner is its modifier. Any category (X) can thus expand with modifiers (Refiners) either before or after a head, within a phase of that same category (Phase X). Head-Modifier Schema [Phase X (Modifier) – X – (Modifier)] Perspective Shifts follow the same head-modifier relationship as Refiners, but instead of honing in on information, they provide a less prototypical spatial viewpoint of the same information. For example, if Figure 5.1 replaced the Refiner with an aerial viewpoint of the same action as in the Initial panel (boxer reaching back to punch the other), this panel would be a Perspective Shift on the prototypical viewpoint (a lateral view) in the prior panel. Both Refiners and Perspective Shifts are ‘modifiers’ in the Head-Modifier Schema above. If a head were to be deleted, the modifier takes the role of its head: if

From Visual Narrative Grammar to Filmic Narrative Grammar

99

the Initial in Figure 5.1 were deleted, the Refiner would become an Initial, without needing a larger phase. The other primary way to expand a sequence repeats narrative categories (Cohn 2015b). Conjunction occurs when a single category (X) repeats within a phase of that category (Phase X): Conjunction schema [Phase X X1 - X2 -… Xn] This notion of ‘conjunction’ is different from the semantic relations often posited in discourse and film theories (Martin 1983; van Leeuwen 1991; Bateman/Schmidt 2012), here aligning with conjunction from syntactic theory: successive panels play the same grammatical role within a constituent

Figure 5.2 Different types of narrative conjunction using the repetition of a single narrative category (Initials) to show various semantic information (actions, characters in a scene, parts of an individual, or semantically associated elements), which could also be framed by a single image.

100

Neil Cohn

of that category. These units can then map to meaning in several ways (Cohn 2015b). Figure 5.2 depicts several potential semantic correspondences to an Initial constituent containing three conjoined panels. Along the lower left tier, each three-panel sequence shows an option for an Initial sequence showing (a) actions or events (A-Conjunction), (b) characters within a scene (E-Conjunction), (c) parts of a single character (N-Conjunction), or (d) disparate semantically associated elements (S-Conjunction). These conjoined images at left create the conceptual equivalent of the panels to the right. That is, the three conjoined images could be replaced using a single, non-conjoined image in the right tier (a diagnostic for assessing conjoined panels). Thus, Figure 5.2 depicts different types of semantic manifestations of the same narrative conjunction.2 Altogether, these three narrative schemas, stored in long-term memory as ‘constructions’, align with the same basic principles of combination found at the syntactic level (Jackendoff 2002; Culicover/Jackendoff 2005). However, human languages do not just use basic abstract combinatorial schemas – there are thousands of constructions that use the basic schemas in regularized ways or depart from those canonical patterns. Similarly, VNG allows for constructional patterns beyond these basic schemas, both using and departing from these basic principles.

4 VNG and Film Theories I have now outlined the basic constructs of VNG as they are proposed for static visual narratives. Among the characteristics of this model are categorical roles played by units and constituents, hierarchic structures that allow connections across distances, modifiers that expand on basic sequencing, and the storage of these elements as constructional patterns in memory. We can now ask: How might these structures relate to those posited in film theories?

4.1 Constructional Patterns in Film Let’s start by looking at Metz’s (1974) grande syntagmatique for the ‘language’ of film. Metz outlines several patterned ways that film shots relate to each other, such as semantic associative relations, alternating shots, and temporally progressive but non-narrative relations, among others. Though posited as articulating syntagmatic relations – i.e., how shots string into sequences – they reflect a more paradigmatic character – i.e., patterned regularities that subsequently appear in sequences (Bateman 2007; Bateman/Schmidt 2012). Subsequently, Metz’s approach has been dissected and reinterpreted as motivated by underlying primitives which can better characterize a range of relationships between shots (Bateman 2007; Bateman/Schmidt 2012). On the one hand, Metzian (and subsequent) relations do characterize connections that extend across sequential shots (syntagmatic), but on the other, they are systematic conventions that comprise specific regularities

From Visual Narrative Grammar to Filmic Narrative Grammar

101

(paradigmatic), not abstract combinatorial ‘rules’. Such tension is precisely what we would expect in grammatical constructions (Goldberg 1995; Jackendoff 2002) – they are sequential patterns that instantiate in the lexicon of a language. They are at once grammatical (syntagmatic) and lexical (paradigmatic). Many of the Metzian patterns incorporate as constructions within VNG, often emerging from mappings between the abstract combinatorial rules and varying semantic meanings. Consider the varying ways that conjunction maps to semantics in Figure 5.2. Metz’s ‘episodic syntagma’ are similar to A-Conjunction, while ‘bracket syntagma’ are similar to S-Conjunction (among others). In VNG, these ‘types’ are recognized as unique patterns, yet all use a common underlying combinatorial principle of conjunction (Cohn 2015b). VNG also provides such constructions with information beyond just the characterization of patterned semantic relations –such patterns play functional roles in the context of a sequence (discussed below). Because VNG is a construction grammar, it also allows novel idiomatic patterns, be they included in Metz’s taxonomy or others (e.g., Branigan 1992; Bateman 2007; Bateman/Schmidt 2012). Because VNG combines constructions in complex ways, other surface patterns may emerge from underlying combinatorial structure. Take for example Metz’s alternating syntagma, also known as ‘crosscutting’ or ‘multitracking’ (Bordwell/Thompson 1997; Bateman/Schmidt 2012), where images flip back and forth between the actions of different characters. Consider Figure 5.3, from the original Star Wars (Lucas 1977), which shows R2-D2 getting captured by droid-selling Jawas. Analyzed by VNG, this particular sequence opens with Establishers, which show R2-D2 rolling through a rocky landscape (shot 1), then shows Jawas watching (2.1) him as they back into the shadows (2.2). These E-Conjoined Establishers set up the characters’ relationship; yet, because a distance separates them, the characters occupy their panels and the phase maps to the inference of the spatial relationship between them (subscript ‘e’).3 As R2-D2 continues rolling (3) the Jawas watch him, (4.1) and dart off (4.2). These E-Conjoined Initials initiate the subsequent conjoined Peaks of R2-D2 (5) getting shot by the Jawas (6.1–3). Conjoined Establishers then set up the new relationship between the electricity striking R2-D2 (7) as the Jawas watch (8). This spreads around the droid (9) in a subsequent Initial and reaches its climax as it surrounds R2-D2 (10) in the main Peak of the sequence, before the electricity dissipates in a Release (11.1), which causes R2-D2 to fall over lifelessly (12.1–2) in the final Release of the broader sequence. This sequence is characterized by, multitracking between R2-D2 and the Jawas. In VNG, each pair of shots constitutes an E-Conjoined constituent leading up to the main climax of the sequence. Though the surface form crosscuts between characters, functionally similar panels group together, potentially operating on different levels – i.e., the opening Establishers are hierarchically higher than the conjunctions embedded within the Initial

102

Neil Cohn

Figure 5.3 Narrative grammar applied to a sequence from Star Wars (12:00–12:27).

constituent. In each case, the local relations between panels map to semantic structure (here not shown in full, but notated with ‘e’) whereby a viewer infers that both characters are in the same spatial location, despite never seeing them together (Cohn 2015b). Notably, the climactic Peak provides the first shot where both Jawas and R2-D2 appear together. This alternation leading to a ‘convergence’ is characteristic of many multitracked patterns (Cohn 2013a). Given the complexity of this analysis, it is worth asking: How might we check if it is correct? Since VNG is committed to integration with empirical verification, analyses emerge through the use of diagnostic tests (deletion,

From Visual Narrative Grammar to Filmic Narrative Grammar

103

Figure 5.4 Paraphrase of the narrative grammar for the Star Wars sequence (1977, 12:00–12:27) in Figure 5.3.

manipulation, or substitution of units) which assess the categories and structure of sequences (see Cohn 2015a for a tutorial), and subsequently provide the basis for psychological experimentation (e.g., Cohn 2014b; Cohn et al. 2014; Hagmann/Cohn 2016). Here we apply one such test to confirm the roles of the broader narrative arc: a paraphrase. Because Peak panels motivate the top-level categories as the ‘head’ of the constituent (i.e., other categories expand from the head Peak), we can drop out all non-Peaks from non-maximal nodes from Figure 5.3 (e.g., within the Initial and Peak constituents) to paraphrase the sequence. This results in Figure 5.4, which depicts these primary shots, providing an adequate paraphrase. Note that the alternation/convergence pattern still maintains, only now simplified to its core elements. Such a test provides evidence that these panels are playing particular functional roles in the sequence, and do so at different levels of structure. Thus, in VNG, the Metzian alternation emerges from interactions between combinatorial structures, namely successive conjunctions. This interaction of structures may also be entrenched as a construction within the narrative grammar (Cohn 2013a). Note also that the successive conjunctions alone do not characterize this sequence: the top level maintains a canonical narrative arc, thereby allowing the lower-level shots and conjunctions to play categorical roles in relation to each other. This organization— facilitated by the parallel structures of narrative and meaning – allows the structure to avoid unnecessary transformational rules to manipulate one ‘deeper’ sequence of separate tracks to become a ‘surface’ linear sequence (Carroll 1980; Buckland 2000), or to require ‘shuffling’ semantic structures into a layout (Bateman/Schmidt 2012). If this ‘ABAB’ alternation is indeed a construction stored in memory, then it should be preferable to others for the same components, as opposed to

104

Neil Cohn

being an emergent pattern resulting from varying equal structural choices. In recent pilot research on drawn visual narratives, we used a forced choice test that asked participants to choose which pattern of sequence they prefer (N=33). Here, 78.5% of participants preferred ABAB alternation patterns to AABB patterns, while 90.5% preferred ABAB patterns to ABBA patterns. Such results imply that preference is given to the alternation pattern over others – at least for drawn visual narratives – suggesting some evidence for a construction stored in memory. In sum, like other approaches, VNG recognizes that alternating shots of characters is a particular patterned structure (Metz 1974; Bateman/Schmidt 2012). Here, it is a surface pattern arising from underlying structures stored in memory as a construction (Cohn 2013a), and able to be modified (e.g., convergence, embedding). Such construction uses interactions between structures for meaning (where the two tracks of events are comprehended) and basic narrative schemas that capture categorical roles of shots/panels relative to each other (where the linear sequence uses multitracking). In some ways this is analogous to the editing process: multiple tracks of film footage (the unconstrained semantics) are edited together to form a linear sequence (narrative). However, in VNG, these separate structures are not external to a comprehender (as in editing), but rather are internal cognitive processes (Cohn et al. 2012; Cohn et al. 2014).

4.2 Hierarchic Structure As shown above, VNG is characterized by hierarchic structures. Experimental evidence of such hierarchies in film structure extend back decades (e.g., Carroll/Bever 1976), though more recently has combined both behavioral and neurocognitive methods (Zacks 2014). For example, participants consistently agree on where to segment short videos of events, and since fine-grained segmentations both align with and fall within coarse-grained segments, it implies a hierarchic organization (e.g., Zacks et al. 2010). In addition, brain activation from passive viewing of the same visual events correlates temporally with the boundaries between events later identified through a segmentation task (Zacks et al. 2011). Such boundaries have been speculated as being caused by the discontinuities in meaning between shots, such as changes in characters, location, or time (Zacks et al. 2010; Magliano/Zacks 2011). This idea is at least somewhat consistent with the structures found in semiotic film theories, which focus on the hierarchies created by dynamically changing aspects of meaning (Bateman/Schmidt 2012; Wildfeuer 2014). In VNG, hierarchy is not dependent on meaning alone, and indeed recent research has shown that narrative categories (discussed below) provide a stronger predictor for participants’ segmentations of a drawn sequence than do changes in meaning (Cohn/Bender 2016). In addition, hierarchy in VNG is not uniform, but rather emerges from different schematic constructions (e.g., the canonical arc, conjunction, and the head-modifier schema),

From Visual Narrative Grammar to Filmic Narrative Grammar

105

and their embedding within each other. For example, narrative categories recursively characterize individual panels/shots, whole constituents, and even plotlines. Properties percolate up to be able to cover both lower- and higher-level structures hierarchically. This is shown in short form between Figures 5.3 and 5.4 – primary structures absorb into a more important head of phases. VNG predicts this to occur at larger levels as well, such that primary Peaks at the shot-level should also provide key information for the plot-level narrative. For example, just as the structures in Figure 5.3 can ‘percolate up’ into Figure 5.4, this entire sequence plays a role within the broader narrative structure of the movie. This sequence is likely one among many embedded Peaks that constitute the Establisher of the overall story (i.e., it shows the Peak of the scene where the Jawas capture R2-D2, which thereby contributes to the ‘set up’ of Luke acquiring the droids, and thus, facilitates his larger adventure). This type of recursion is perhaps more apparent when it uses obvious structured patterns, and similar observations of multilevel recursive patterning are made for Metz’s patterns specifically (Fledelius 1978). For example, the final, climatic sequence of Star Wars shifts between scenes of the starfighters attempting to blow up the Death Star and the Rebellion fearing destruction at its hands. This multitracking does not occur at the shot level, but rather at the scene level. Thus, in addition to constructions manifesting as a surface structure to a sequence, VNG can capture observations about hierarchy at different levels.

4.3 Categorical Roles Film (and narrative) theory has long recognized that plotlines are organized into narrative ‘acts’ that function in different ways. Such observations extend back to Aristotle’s three-act structure and Freytag’s (1894) five-act ‘pyramid’ for theatre, and, more recently, in a four-act structure theorized for the plots of mainstream Hollywood movies (Thompson 1999; Bordwell 2006). Similar structures have appeared in many models of narrative for both verbal, visual, and filmed narratives (for review, see Cohn 2013b; Cutting 2016). Recent work has also shown that such plot-level structures are characterized by features of surface elements like shot pacing, cuts, and even motion and brightness in shots (Cutting 2016). Such narrative roles are posited as psychologically reducing the processing costs of following a story (Cutting 2016) and contribute towards visual narratives being distinguished from the dynamic updating of semantic information which could just as readily characterize the perception of everyday experience (Radvansky/Zacks 2014; Zacks 2014). Insights of narrative roles throughout plotlines are maintained in VNG, and some preliminary work has suggested that VNG is effective for characterizing the shots of narrative data videos (Amini et al. 2015). As discussed above, the narrative categories in VNG also extend down to the level of

106

Neil Cohn

scenes and shots, in addition to broader plotlines. This hierarchic nature allows VNG to capture additional observations about functional roles. Consider an establishing shot (Bordwell/Thompson 1997; Bordwell 2007; Cutting/Iricinschi 2015), which is a long shot showing an expansive view on both the environment and the people in it. It functions to situate elements of a scene before actions taking place, though experiments suggest that its absence does not necessarily impair the comprehension of a sequence (Kraft/ Cantor/Gottdiener 1991). Establishing shots are clearly similar to Establishers in VNG, which also are rarely noticed when missing (Cohn 2014b). However, the notion of an establishing shot is more limiting that that of Establishers: they only account for shots that use a wide perspective on the whole scene. Numerous images or shots may serve this same function, each showing only a glimpse of the overall ‘established’ scene. For example, in Star Wars when Luke enters the Mos Eisley cantina, numerous shots show various aliens in acts of disrepute—all establishing the broader environment. Such a sequence is covered by Metz’s patterns, but simply noting this as a relational syntagma misses the fact that these independent shots provide the same function as would a single establishing shot, only here distributed across multiple shots. VNG would label all of these as Establishers, but conjoined together using E-Conjunction. In addition, in VNG, such a Metzian surface pattern (E-Conjunction) could manifest in other conjoined narrative categories. We might, therefore, think of establishing shots as ‘prototypical single unit Establishers’, though not all Establishers might be used as establishing shots. Thus, in sum, because of its hierarchic, recursive properties, VNG allows us to characterize narrative roles at levels of the shot, scene, or plotline. Further, basic modifiers in VNG like conjunction characterize functional similarities across different shots/syntagma. Such features allow VNG to capture numerous features of film theories incorporated directly within the basic properties of the model.

5 VNG and Filmic Narrative Grammar Finally, we can now ask: how does VNG change when describing moving images versus static images? There are many clear differences between static visual narratives, like those found in comics, and moving visual narratives, like those found in films. A basic prediction may be that: should differences arise between the structure of static and moving image sequences, they would be a direct result of the differences in the modalities themselves (Cohn 2013a). We now turn to exploring such potential differences.

5.1 Form and Vocabulary First, obvious differences emerge because static visual narratives are drawn while films are usually captured by a camera. Drawings are a trait of human

From Visual Narrative Grammar to Filmic Narrative Grammar

107

biology – all children draw, whether it is using tools (pens, crayons) or just their fingers (in sand, paint, etc.), though tempered by cultural exposure to graphic patterns, just like language (Wilson 1988; Cohn 2012). Thus, drawn visual narratives manifest using a systematized graphic structure of patterned lines and shapes. Film does not use such producible patterns of an expressive modality (drawing), but rather is mediated by technology that is required for both its production (creating film) and reception (viewing film). In this way, film does not require storage of graphic patterns, but rather bypasses it for a direct interface to general perception (except in animation, discussed below). Despite this, it is worth remembering that most filmic sequences usually begin as drawn sequences: films are often preceded by storyboards which are static sequential images comparable to those found in comics. Thus, the production of meaningful image sequences in film usually uses static visual sequences as their basis, though their end-state emerges with perceptual motion within and between these static units, along with multimodal information of music, speech, etc.4 Because of this, the narrative structure of film uses an inherently hybrid form. Film captures perceptual information through a camera; this alone would be comprehended through general perceptual principles and semantic understandings related to event knowledge. This information is then broken up into shots and edited together using a film-maker’s cognitive combinatorial narrative principles – here posited as Visual Narrative Grammar (often first produced in storyboards) – such that those units progress in a coherent sequence. The resulting ‘Filmic Narrative Grammar’ is thus a hybrid between the natural VNG and its manifestation using units of general perception. Unlike drawing, film’s hybrid status is not natural. Static drawings come from the universal human ability to create meaning graphically. Instead, film captures perceptual information from an external source by filming it with a specific tool – a camera – and is thus mediated by technology developed only recently. FNG moves one step closer to VNG in animation, where the units are once again drawn. However, animation with motion may change VNG just as FNG does, by virtue of the affordances of moving images (discussed below). One interesting aspect of this difference is that contiguous motion does not bind static drawn narratives across images (Zacks/Magliano 2011; Berliner/Cohen 2001; Smith 2012). Rather, drawn images often use discontinuous images where no ‘smooth motion’ is carried from one image to the next. For example, a scene may show a person walking behind a tree and coming out the other side. Continuity editing would cut the scene when the person disappears behind the tree so that their emergence from the other side looks fluid and continuous, but a discontinuous edit might stop the first shot before the person reaches the tree, only to cut to them coming out the other side.

108

Neil Cohn

In film, such discontinuity between shots alone should be jarring and less coherent because it deviates from our expectation of perceptual continuity (Zacks/Magliano 2011; Berliner/Cohen 2011; Smith 2012). Perhaps more discontinuous image relations can persist in drawn narratives because they select only discrete moments out of an inferred continuous action and are not tied to perceptual motion – only conceptual motion. Or, perhaps, the discontinuity between actions is actually acceptable in drawn form, but that film continuity gives a ‘bonus’ for comprehension by further tying them together through motion? Alternatively, perhaps this is modality specific: Maybe discontinuity is acceptable only for the drawn form, but general perception in film leads to an expectation of continuity? This would predict that discontinuous shots would be more acceptable in animation, and that lack of continuity would be less acceptable for static narratives using general perception, as in ‘photo comics’ which are not drawn. While discontinuous shots likely do violate the perceptual continuity in film (as has been claimed), the relations to static, drawn narratives are worth exploring through empirical testing. Insofar as film uses general percepts, it usually lacks the ‘vocabulary’ of drawn visual narratives. For example, the visual language used in comics uses many conventionalized patterns like motion lines to imply the path of moving objects, bubbles and balloons to depict thoughts and speech, star-shaped flashes to indicate impacts, and many other patterns, which are often culture specific (Cohn 2013a). In general, films do not use these conventions unless they are animated or trying to reference the style of comics (as in the film adaptation of the comic Scott Pilgrim vS. the World (Wright 2010)). Another dimension of difference between filmed and drawn narratives, at least those in comics, is the layout of panels on a page. Filmed narratives typically involve no physical juxtaposition of shots, with images proceeding temporally in the single space of the screen. It is true; some films do play with the screen space to divide it up into component ‘panels’ through split screens or more complicated framings, as in Ang Lee’s Hulk (2003; cf. Bateman/Veloso 2013). However, ‘filmic panels’ and page layouts in comics differ in important ways. First, the content in filmic panels usually occurs simultaneously in real time, thereby implying narrative simultaneity, while those in static form may or may not be at simultaneous moments based on the particulars of the sequence. Second, there is no ‘reading order’ for filmic panels on a screen independent of the relations between their content (in part due to their simultaneity and internal motion); ordering may be motivated by a variety of factors, including when those filmic panels appear, how they move around on the screen and other aspects of internal content such as sound. In contrast, the ordered navigation of static images is integral to their comprehension, yet such ordering is provided by a rule system that is external to the content within the frames (Cohn 2013a; Cohn/ Campbell 2015).

From Visual Narrative Grammar to Filmic Narrative Grammar

109

5.2 Narrative Structure Besides graphic structure, the clearest difference between drawn and filmic narratives is that film uses motion while drawings do not. This means that film is a continuous, inherently temporally progressing medium, while drawings form discrete units where the passage of time is ambiguous. For example, Figure 5.2b depicts three different characters in a sequence of conjoined Initials. In drawn form, nothing indicates the passage of time, and indeed, all panels exist at the same narrative state (Initials), which further suggests simultaneity. Without explicit cues in the images, we cannot tell whether time changes. In contrast, if this sequence appeared in film, these shots would all inherently extend across a span of time, and thus, would be more suggestive of time passing between them as well. Additional consequences of motion may impact the units of the grammar. Because a camera can just be left recording, with no cuts, this temporality should be the equivalent of a general perception of events. Such an ongoing recording of information alone has no narrative segmentation (discussed further below), though it may be comprised of perceived events, which do have their segmental structure (Zacks et al. 2011) that interface with the narrative. When that continuous perception is broken up into shots and edited together, the shots take on narrative categories linked through principles of FNG. However, because the units are now in motion, this temporality may “gloss over” features that in the static form would be individuated categories. Thus, the more cuts, and the less event information per cut, the more narrative units will be used in a film. Consider shots 2, 4, 6, 11, and 12 in Figure 5.3. In all of these cases, I have chosen to show the shots by using multiple static frames. This is because the internal structure of these shots uses movement: in shot 2 the Jawas recede into shadows, in 4 they dart away in the shadows, in 6 a Jawa leaps up and shoots a blaster, and in 12 R2-D2 lurches back and falls over. In the tree structure in Figure 5.3, each shot has been notated for a narrative category wherein that movement occurs (e.g., shot 12 is a Release). If we were to consider Figure 5.3 as depicted – as static images and not as continuous shots – each image would have its “have its own narrative category. For example, 12.1 would be an Initial of R2-D2’s starting state before falling in the Peak of 12.2, which would then be contained within a Release constituent. As stated above, continuous shots contain no internal narrative structure, only event structures. Thus, the motion of film shots may ‘gloss over’ what could otherwise be individuated units of the grammar in the drawn form, instead of mapping to a higher level of structure. Not only do the elements within a film shot move (i.e. characters and objects move around) but so can the camera itself, with panning and zooming. For example, panning allows a camera to navigate through a space while only showing a portion at a time, or retaining focus on certain moving elements in that space. Drawings can attempt to simulate panning with

110

Neil Cohn

Figure 5.5 Polymorphic divisional panel of a bee flying.

or without continuous motion, such as ‘divisional’ panels that divide a singular background into different panels (Cohn 2014a). Figure 5.5 shows divisional panels that depict a moving bee. In graphic form, this illusion of motion will always retain some segmentation by the fact that the units do not inherently move. Another example is zooming. A person may be shown in full body, only to have the camera zoom into a close-up of their eyeball in one continuous shot. In static form, a single panel of a full body followed by a close up panel on their eye would be a Refiner. However, in film, the continuity of the zoom makes the ‘head’ category bleed into the ‘Refiner’. Imagine if a shot zoomed from a fully body shot into a close up of an eye, then back to the full body. Would the zoom retain a ‘special’ status as a Refiner (and what would that thereby do the previous state?), or would the continuous motion merely gloss over such structure as it does for the continuous events? A similar issue occurs with Perspective Shifts, which merely alter the viewpoint on a scene from one image to another. The famous ‘bullet-time’ scene from The Matrix (Wachowski/Wachowski 1999) captures this well: the camera sweeps in a circle around the frozen character of Neo, showing the whole scene from 360°. In graphic form, this would require several discrete Perspective Shift panels, but in film it would at least be possible to accomplish with a single shot. Another example applies to the construction of an environment. In the second shot immediately following the end of the scene in Figure 5.3, the camera pans from R2-D2 lying on the ground to the standing Jawas, leaving R2-D2 out of the frame. As in the E-Conjunction in shots 7 and 8 in Figure 5.3, R2-D2 and the Jawas are never shown on screen at the same time. However, the continuity of this single film shot (and the background) likely ‘dampens’ the inferential demand compared to that in unitized panels/ shots. In film, this depicted contiguity between characters shows us that they belong to a single environment. With discrete images, this must be entirely

From Visual Narrative Grammar to Filmic Narrative Grammar

111

inferred or must use a ‘divisional’ panel as in Figure 5.5. Thus, for all aspects of the narrative grammar, from basic narrative roles to modifiers, the introduction of motion can mitigate or wash over what in static form might be clearly discrete units. Thus far, I have claimed that the continuous motion of film leads FNG to ‘gloss over’ structure that would otherwise be individuated in VNG when drawn. A strong view would thus posit that cuts between shots are necessary to individuate the components of the narrative grammar. Without such divisions, a shot merely shows the progression of event structure, but not narrative. An alternative perspective though might say that cuts are not necessary to divide components, and event structures can map to narrative states within moving shots. This would be consistent with psychological theories of event cognition, which posit the same segmental structures within everyday events as those signaled by filmic cuts (Zacks/Magliano 2011; Radvansky/Zacks 2014; Zacks 2014), but which do not differentiate event knowledge from narrative structures (as in VNG). Narrative structures must appear in some continuous percepts, or else no sense of narrative would emerge from live theatre other than hard scene breaks. However, this view would then need to explain the difference between general perception of everyday events and filmed narratives – which are clearly not the same phenomenon. Conversely, not all cuts must constitute cues for narrative structure. For example, the filming of live events like sports, concerts, political debates, and others also use cuts to change camera angles, show different people, etc. Each cut within such filming does not signal a narrative break – indeed, such events are usually not narratives at all! Thus, cuts alone are not signals for narrative categories, since shots may not even have narrative categories. There may also be a middle ground in the case of narrative films. For example, perhaps narrative structures – which are different from just event knowledge (Cohn 2015b) – appear in continuous shots only when they depart from general percepts or use cues to signal the changing between narrative units. Thus, there would be no component narrative categories for simple continuous events (i.e., in Figure 5.3, shot 12 would not divide into an Initial-Peak constituent), but such structure would arise for continuous zooms for Refiners or panning for Perspective Shifts (like the Matrix’s bullet-time). These are conventions of films that do not occur in everyday perception, and thus, would be ‘marked’ as necessitating narrative structures. Films may also use transitions to tell us that they are continuously shifting from one narrative category to another. Imagine an expansive establishing shot of characters in a scene (an Establisher), which then zooms in one motion to the characters involved in actions within that scene (Initial, etc.). In this case, the zoom may functionally tell us about the change from one category to another, without a cut. Disentangling these issues remains an empirical question.

112

Neil Cohn

5.3 Summary Overall, this section has outlined several ways in which the structure of static, drawn narratives differs from that of moving, filmic narratives. Such departures are hypothesized as primarily emerging from the differences in basic properties of the media themselves (static vs. moving). Some of these differences are summarized in Table 5.1. Table 5.1 Gross differences in dimensions between prototypical cases of drawn and filmed narratives Static, drawn narratives Production is a biologically based human ability (drawing) Uses patterned graphic schema for both iconic and symbolic elements (i.e., stored lines and shapes in a visual vocabulary) Static content in images Static depictions in images Ambiguous temporality between units unless otherwise depicted Spatial juxtaposition of units (in page layout) requiring non-content based navigational rules

Moving, filmic narratives Production is technologically mediated (non-natural) Uses general perception (not a patterned visual vocabulary, with the exception of animation) Moving content in film Moving camera in film (panning, zooming) Pervasive sense of temporality between units because of ongoing temporality of motion Temporal juxtaposition of units unfurling on a screen. When spatially juxtaposed frames appear on a screen, they involve no independent navigational rules.

6 Conclusion Altogether, I have introduced a theory of narrative grammar that includes numerous insights made by previous theories of visual narratives, such as narrative categories, hierarchic structures, and patterned constructions. In its application to static visual narratives, this architecture has growing experimental support for its psychological validity. Insofar as the brain relies on domain-general mechanisms for narrative comprehension, such an approach has been posited as capable of describing film comprehension as well, and work has begun to examine such applicability (e.g., Amini et al. 2015). However, to the extent that static and moving narratives create different affordances, this narrative grammar alters in structure for the hybrid nature of film’s moving images compared to the static, more natural drawn modality. This theory thus allows for specific predictions about – and the ability to test – both the comprehension of FNG and the differences between VNG and FNG through empirical investigation.

From Visual Narrative Grammar to Filmic Narrative Grammar

113

Notes 1. Note: some categories such as Prolongations and Orienters are omitted for simplicity (Cohn 2013b). 2. Note that the framing categories listed on the right of Figure 5.2 (macro, mono) do not precisely map to ‘shot types’ in film. While a prototypical macro may use a long shot, macros are defined simply by the interaction of multiple characters in a frame. Mono panels likewise depict only a single character, whether shown in a long, full, or close shot (Cohn 2013a). Thus, these categories are about quantity of information, not how that information is presented. 3. The ‘e’ notation is shorthand. In actuality, the narrative structure for VNG is accompanied by additional components from a conceptual/event structure and a spatial structure, which specify these semantic relations explicitly. The ‘e’ actually reflects the interface between the conceptual and spatial structures with those in the narrative structures (Cohn 2015b), here omitted for simplicity. 4. Note that the ‘parallel architecture’ approach in which VNG is embedded is also ideal for dealing with the multimodal characteristics of film. Though not focused on here, this model allows different structures to independently operate, yet to coalesce to create emergent structures and meaning not found in any one modality (Cohn 2016).

Bibliography Amini, F./Riche, N.H./Lee, B./Hurter, C./Irani, P. (2015). “Understanding Data Videos: Looking at Narrative Visualization through the Cinematography Lens.” In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, pp. 1459–1468. Bateman, J.A. (2007). “Towards a Grande Paradigmatique of Film: Christian Metz Reloaded.” Semiotica 167, pp.13–64. Bateman, J.A./Schmidt, K.-H. (2012). Multimodal Film Analysis: How Films Mean. New York: Routledge. Bateman, J.A./Veloso, F.O.D. (2013). “The Semiotic Resources of Comics in Movie Adaptation: Ang Lee’s Hulk (2003) as a Case Study.” Studies in Comics 4(1), pp. 135–157. DOI: 10.1386/stic.4.1.135_1. Bateman, J.A./Wildfeuer, J. (2014). “A Multimodal Discourse Theory of Visual Narrative.” Journal of Pragmatics 74, pp. 180–208. DOI: 10.1016/j. pragma.2014.10.001. Berliner, T./Cohen, D.J. (2011). “The Illusion of Continuity: Active Perception and the Classical Editing System.” Journal of Film and Video 63(1), pp. 44–63. Bordwell, D. (2006). The Way Hollywood Tells It: Story and Style in Modern Movies. Berkeley: University of California Press. Bordwell, D. (2007). Poetics of Cinema. New York: Routledge. Bordwell, D./Thompson, K. (1997). Film Art: An Introduction. 5th Edition. New York: McGraw-Hill. Branigan, E. (1992). Narrative Comprehension and Film. London: Routledge. Buckland, W. (2000). The Cognitive Semiotics of Film. Cambridge: Cambridge University Press. Byram, M.L./Garforth, C. (1980). “Research and Testing Non-Formal Education Materials: A Multi-Media Extension Project in Botswana.” Educational Broadcasting International 13(4), pp. 190–194.

114

Neil Cohn

Carroll, J.M. (1980). Toward a Structural Psychology of Cinema. The Hague: Mouton. Carroll, J.M./Bever, T.G. (1976). “Segmentation in Cinema Perception.” Science 191(4231), pp. 1053–1055. Chateau, D. (1987). La Cinéma Comme Langage. Brussels: AISS, Publications de la Sorbonne. Chomsky, N. (1965). Aspects of the Theory of Syntax. Cambridge: MIT Press. Cohn, N. (2010). “The Limits of Time and Transitions: Challenges to Theories of Sequential Image Comprehension.” Studies in Comics 1(1), pp. 127–147. Cohn, N. (2012). “Explaining “I Can’t Draw”: Parallels between the Structure and Development of Language and Drawing.” Human Development 55(4), pp. 167–192. DOI: 10.1159/000341842. Cohn, N. (2013a). The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images. London: Bloomsbury. Cohn, N. (2013b). “Visual Narrative Structure.” Cognitive Science 37(3), pp. 413–452. DOI: 10.1111/cogs.12016. Cohn, N. (2014a). “The Architecture of Visual Narrative Comprehension: The Interaction of Narrative Structure and Page Layout in Understanding Comics.” Frontiers in Psychology 5, pp.1–9. DOI: 10.3389/fpsyg.2014.00680. Cohn, N. (2014b). “You’re a Good Structure, Charlie Brown: The Distribution of Narrative Categories in Comic Strips.” Cognitive Science 38(7), pp. 1317–1359. DOI: 10.1111/cogs.12116. Cohn, N. (2015a). “How to Analyze Visual Narratives: A Tutorial in Visual Narrative Grammar.” Online: http://www.visuallanguagelab.com/P/VNG_Tutorial.pdf [last accessed: 1 March 2016]. Cohn, N. (2015b). “Narrative Conjunction’s Junction Function: The Interface of Narrative Grammar and Semantics in Sequential Images.” Journal of Pragmatics 88, pp. 105–132. DOI: 10.1016/j.pragma.2015.09.001. Cohn, N. (2016). “A Multimodal Parallel Architecture: A Cognitive Framework for Multimodal Interactions.” Cognition 146, pp. 304–323. DOI: 10.1016/j. cognition.2015.10.007. Cohn, N./Bender, P. (2016). “Drawing the Line between Constituent Structure and Coherence Relations in Visual Narratives.” Journal of Experimental Psychology: Learning, Memory, & Cognition (forthcoming). Cohn, N./Campbell, H. (2015). “Navigating Comics II: Constraints on the Reading Order of Page Layouts.” Applied Cognitive Psychology 29, pp. 193–199. DOI: 10.1002/acp.3086. Cohn, N./Jackendoff, R./Holcomb, P.J./Kuperberg, G.R. (2014). “The Grammar of Visual Narrative: Neural Evidence for Constituent Structure in Sequential Image Comprehension.” Neuropsychologia 64, pp. 63–70. DOI: 10.1016/j. neuropsychologia.2014.09.018. Cohn, N./Kutas, M. (2015). “Getting a Cue before Getting a Clue: Event-Related Potentials to Inference in Visual Narrative Comprehension.” Neuropsychologia 77, pp. 267–278. DOI: 10.1016/j.neuropsychologia.2015.08.026. Cohn, N./Kutas, M. (in preparation). “What’s your neural function, visual narrative conjunction? Grammar, meaning, and fluency in sequential image processing.” (forthcoming). Cohn, N./Paczynski, M. (2013). “Prediction, Events, and the Advantage of Agents: The Processing of Semantic Roles in Visual Narrative.” Cognitive Psychology 67(3), pp. 73–97. DOI: 10.1016/j.cogpsych.2013.07.002.

From Visual Narrative Grammar to Filmic Narrative Grammar  115 Cohn, N./Paczynski, M./Jackendoff, R./Holcomb, P./Kuperberg, G. (2012). “(Pea) nuts and Bolts of Visual Narrative: Structure and Meaning in Sequential Image Comprehension.” Cognitive Psychology 65(1), pp. 1–38. DOI: 10.1016/j. cogpsych.2012.01.003. Cohn, N./Wittenberg, E. (2015). “Action Starring Narratives and Events: Structure and Inference in Visual Narrative Comprehension.” Journal of Cognitive Psychology 27(7), pp. 812–828. DOI: 10.1080/20445911.2015.1051535. Cohn, Neil, and Patrick Bender. 2016. “Drawing the line between constituent structure and coherence relations in visual narratives.” Journal of Experimental Psychology: Learning, Memory, & Cognition 42 (8). Colin, M. (1995). “The Grande Syntagmatique Revisited.” In: Buckland, W. (ed.): The Film Spectator: From Sign to Mind. Amsterdam: Amsterdam University Press, pp. 45–86. Culicover, P.W./Jackendoff, R. (2005). Simpler Syntax. Oxford: Oxford University Press. Cutting, J. E. (2016). “Narrative theory and the dynamics of popular movies.” Psychonomic Bulletin & Review 1-31. doi: 10.3758/s13423-016-1051-4. Cutting, J. E./Iricinschi, C. (2015). “Re-Presentations of Space in Hollywood Movies: An Event-Indexing Analysis.” Cognitive Science 39(2), pp. 434–456. DOI: 10.1111/cogs.12151. Eisenstein, S. (1942). Film Sense. Translated by J. Leyda. New York: Harcourt, Brace World. Fledelius, K. (1978). “Syntagmatic Film Analysis – With Special Reference to Historical Research.” Untersuchungen zur Syntax des Films I, pp. 31–68. Freytag, G. (1894). Technique of the Drama. Chicago: S.C. Griggs & Company. Fussell, D. /Haaland, A. (1978). “Communicating with Pictures in Nepal: Results of Practical Study Used in Visual Education.” Educational Broadcasting International 11(1), pp. 25–31. Gernsbacher, M.A. (1985). “Surface Information Loss in Comprehension.” Cognitive Psychology 17, pp. 324–363. Goldberg, A. (1995). Constructions: A Construction Grammar Approach to Argument Structure. Chicago: University of Chicago Press. Hagmann, C. E./Cohn, N. (2016). “The Pieces Fit: Constituent Structure and Global Coherence of Visual Narrative in RSVP.” Acta Psychologica 164, pp. 157–164. DOI: 10.1016/j.actapsy.2016.01.011. Ildirar, S./Schwan, S. (2015). “First-Time Viewers’ Comprehension of Films: Bridging Shot Transitions.” British Journal of Psychology 106(1), pp. 133–151. DOI: 10.1111/bjop.12069. Jackendoff, R. (1990). Semantic Structures. Cambridge: MIT Press. Jackendoff, R. (2002). Foundations of Language: Brain, Meaning, Grammar, Evolution. Oxford: Oxford University Press. Kraft, R.N./Cantor, P./Gottdiener, C. (1991). “The Coherence of Visual Narratives.” Communication Research 18(5), pp. 601–616. Kunzle, D. (1973). The History of the Comic Strip. Vol. 1. Berkeley: University of California Press. Levin, D. T./Simons, D.J. (2000). “Perceiving Stability in a Changing World: Combining Shots and Integrating Views in Motion Pictures and the Real World.” Media Psychology 2(4), pp. 357–380. Magliano, J.P./Larson, A.M./Higgs, K./Loschky, L. (2015). “The Relative Roles of Visuospatial and Linguistic Working Memory Systems in Generating Inferences

116  Neil Cohn during Visual Narrative Comprehension.” Memory & Cognition, pp.1–13. DOI: 10.3758/s13421-015-0558-7. Magliano, J.P./Zacks, J.M. (2011). “The Impact of Continuity Editing in Narrative Film on Event Segmentation.” Cognitive Science 35(8), pp. 1489–1517. DOI: 10.1111/j.1551-6709.2011.01202.x. Mandler, J.M./Johnson, N.S. (1977). “Remembrance of Things Parsed: Story Structure and Recall.” Cognitive Psychology 9, pp. 111–151. Marslen-Wilson, W.D. (1987). “Functional Parallelism in Spoken ­Word-Recognition.” Cognition 25(1-2), pp. 71–102. Martin, J. R. (1983). “Conjunction: The Logic of English Text.” In: Petöfi, J.S./Sözer, E. (eds.). Micro and Macro Connexity of Discourse (= Papers in Textlinguistics 45). Hamburg: Helmut Buske Verlag, pp. 1–72. McCloud, S. (1993). Understanding Comics: The Invisible Art. New York: Harper Collins. Metz, C. (1974). Film Language: A Semiotics of the Cinema. Translated by Michael Taylor. New York: Oxford University Press. Miller, G.A./Isard, S. (1963). “Some Perceptual Consequences of Linguistic Rules.” Journal of Verbal Learning and Verbal Behavior 2(3), pp. 217–228. DOI: 10.1016/s0022-5371(63)80087-0. Osterhout, L./Nicol, J.L. (1999). “On the Distinctiveness, Independence, and Time Course of the Brain Responses to Syntactic and Semantic Anomalies.” Language and Cognitive Processes 14(3), pp. 283–317. Radvansky, G.A./Zacks, J. (2014). Event Cognition. Oxford: Oxford University Press. Schwan, S./Ildirar, S. (2010). “Watching Film for the First Time: How Adult Viewers Interpret Perceptual Discontinuities in Film.” Psychological Science 21(7), pp. 970–976. DOI: 10.1177/0956797610372632. Smith, T.J. (2012). “The Attentional Theory of Cinematic Continuity.” Projections 6(1), pp. 1–27. DOI: http://dx.doi.org/10.3167/proj.2012.060102. Thompson, K. (1999). Storytelling in the New Hollywood: Understanding Classical Narrative Technique. Cambridge: Harvard University Press. van Leeuwen, T. (1991). “Conjunctive Structure in Documentary Film and Television.” Continuum 5(1), pp. 76–114. DOI: 10.1080/10304319109388216. Wildfeuer, J. (2014). Film Discourse Interpretation: Towards a New Paradigm for Multimodal Film Analysis. London/New York: Routledge. Wilson, B. (1988). “The Artistic Tower of Babel: Inextricable Links Between ­Culture and Graphic Development.” In: Hardiman, G.W./Zernich, T. (eds.). ­Discerning Art: Concepts and Issues. Champaign: Stipes Publishing Company, pp. 488–506. Wilson, B. (2016). “What Happened and What Happened Next: Kids’ Visual Narratives across Cultures.” In: Cohn, N. (ed.). The Visual Narrative Reader. London: Bloomsbury, pp. 185–227. Wilson, B./Wilson, M. (1987). “Pictorial Composition and Narrative Structure: Themes and Creation of Meaning in the Drawings of Egyptian and Japanese Children.” Visual Arts Research 13(2), pp. 10–21. Zacks, J.M. (2014). Flicker: Your Brain on Movies. Oxford: Oxford University Press. Zacks, J.M./Braver, T.S./Sheridan, M.A./Donaldson, D.I./Snyder, A.Z./Ollinger, J.M./ Buckner, R.L./Raichle, M.E. (2001). “Human Brain Activity Time-Locked to Perceptual Event Boundaries.” Nature Neuroscience 4(6), pp. 651–655.

From Visual Narrative Grammar to Filmic Narrative Grammar  117 Zacks, J.M./Magliano, J.P. (2011). “Film, Narrative, and Cognitive Neuroscience.” In: Melcher, D.P./Bacci, F. (eds.). Art and the Senses. New York: Oxford University Press, pp. 435–454. Zacks, J.M./Speer, N.K./Swallow, K.M./Maley, C.J. (2010). “The Brain’s ­Cutting-Room Floor: Segmentation of Narrative Cinema.” Frontiers in Human Neuroscience 4, pp. 1–15. Zwaan, R.A./Radvansky, G.A. (1998). “Situation Models in Language Comprehension and Memory.” Psychological Bulletin 123(2), pp. 162–185.

Filmography Matrix, The (1999). Lana Wachowski/Lilly Wachowski. Warner Brothers. USA. Star Wars (1977). George Lucas. Lucasfilm. USA.

6

From Text to Recipient Pragmatic Insights for Filmic Meaning Construction Janina Wildfeuer

1 Introduction In his introduction to an edited volume on the film spectator as an important interface between the film itself as shown on screen and the interpretation by recipients (cf. Buckland 1995), Thomas Elsaesser points out that “one of the major insights of film semiology has been the fact that the ‘meaning’ of film cannot be adequately determined by its formal syntactic and semantic characteristics alone. The interpretation of a film is equally a function of its social context, defined as the set of rules and conventions that spectators in given circumstances bring to bear on the particular films they see.” (Elsaesser 1995: 16) This interest in a more pragmatic and communication-in-context approach to film has first been brought into the focus of film studies mostly within the branches of the so-called ‘second semiology’ or post-structural film theory (cf. Buckland 1995a: 18–19; Simons 1995: 209), concentrating on the relation of the filmic material to its interpreters and thus “marked by a shift in focus from the film as a ‘textual system’ to the ‘role of the spectator’” (Simons 1995: 209). Whereas the era of classical semiology in the 1960s and 1970s rather oriented towards the application of structural linguistics to film analysis, mostly (ineffectively) searching for smallest units and meaningful entities, new developments in the early 1980s oriented towards contextual details and the environment of films as influencing factors for their interpretation. The role of the spectator in this environment has since then been addressed from various perspectives, seeing him/her as being inscribed in the filmic text (as is the case in enunciation theories, cf. Metz 1991; Casetti 1983a, 1983b), as being programmed to develop certain attitudes and responses (as in speech act theoretical approaches to film, for example; cf. Dagrada 1987, Kobow 2007), or as being an active part of the meaning-construction process guided by contextual information and cultural and social constraints (as in Odin’s 1983, 1995 semio-pragmatic approach; see also Simons 1995 for further details).

From Text to Recipient

119

In particular the latter perspective as “a broad theory of pragmatics enriched and completed by the interlocutor’s inferential activities” (Buckland 1995: 21) has recently become more relevant not only for contemporary film analytical approaches (cf., e.g., Bateman 2007; Desilla 2010; Janney 2012; Bateman/Schmidt 2012), but also for more general theories of meaning construction in media artifacts (cf. Sachs-Hombach 2001; Bucher 2007; Bredekamp 2010; Björkvall 2012). The latter is often similarly confronted with the problem of not being able to draw on an adequate semantic description of meaning conveyed by the various semiotic resources in media texts. They then rather ask how recipients as the central element of interpretation construct meaning with the help of contextual knowledge and cultural and social constraints or conventions, thus focusing on information that is not always inherent in the material with which they are dealing. This concentration on the spectator and his/her interpretative operations indeed follows general advancements of the pragmatic turn in the humanities and their focus on dynamic and contextual interpretation, which is an indispensable expansion of all theoretical thoughts on the analysis of communicative artifacts. Nevertheless, this focus also involves – on the other hand and to a certain extent – a shift away from the text itself as the originally primary unit of analysis. This is not only indicated in Simon’s formulation given above but also observable with a closer look at, for example, discussions about the spectator as playing the leading role in film interpretation. Following Odin’s work, for instance, the spectator is a constructed entity whose “production of meaning is entirely based on external determinations” (Odin 1995: 215, emphasis in original). As a consequence, meaning is no longer an intrinsic part of the artifact, but rather seen as an external attribute resulting from the spectator’s active reception. This description goes much further than describing meaning as not being fixed in advance and bringing the role of the recipient into play. Instead, it allocates to the recipient the main and most important function in the meaning-making process and at the same time degrades the role of the text in this process. As Buckland (2000: 76) points out, the “truism that texts cannot determine their own reading [is] a fundamental premise” in Odin’s work and leads to the fact that any possibility of textual analysis is questioned in this context (cf. Buckland 2000: 72). In this regard and despite their linguistic background, the pragmatic aspect in this and also other approaches (cf. Buckland 2000: 52–76) tends to reject a focus on the text itself in favor of external aspects of interpretation. Dramatically, Buckland even highlights that a pragmatic theory of meaning might be “premised on non-communication”, no longer seeing any notion of semantics and artifact- or message-based information as useful for the process of meaning construction, but rather focusing on the recipients’ cognitive operations and the contextual as well as cultural constraints guiding these operations (cf. Buckland 2000: 80–81).1 Instead of approaching filmic and other artifacts with the traditional semiotic notion of codes that regulate and determine meaning in a closed circuit (following Saussure’s (1959 [1915]) fundamental

120

Janina Wildfeuer

discussion of the circuit of speech), the focus then lies on the recipients’ inferences as those mechanisms that make it possible to describe the cognitive operations in further detail (cf. Buckland 2000: 82; Bordwell 1989). As mentioned above, the interest in and attention to inferences has indeed grown considerably in recent works on film and media analysis, not only concerning the immense developments in disciplines such as psychology and neurocognition but also already from a cognitive-semiotic perspective pursued since the 1980s. Buckland himself, for example, takes the pragmatic and pragma-semiotic perspective on film theory as a starting point to bring together film semiotics and cognitive theory “with the objective of modeling filmic competence – that is, the spectator’s knowledge or intuitions about filmic meaning” (Buckland 2000: 141). In his work, the notion of text does not play a significant role and is, as outlined above, sometimes rather questioned in favor of, for example, a grammatical approach following Chomsky’s transformational-generative linguistics. Contrasting approaches go back to the fundamental concepts of abductive reasoning and active semiosis that have already been described in the work of Charles Sanders Peirce (1931–1958) and which take into account the recipient and his/her cognitive operations as well as the material or text as the main object of description and its semiotic resources and structures. For film theory, Bordwell’s approach of making meaning on the basis of inferences (1989) is clearly one of the pioneering works, though not providing a very detailed account of how these inferential processes can be described. A further development in the last few years with regard to the study of inferences in media artifacts can be observed within approaches coming from more contemporary linguistics, including again and pushing forward the notions of text and discourse and re-orienting towards the basic and inherent meaning-making patterns of verbal and non-verbal discourses. Our work in this area follows basic advances in both discourse semantics and multimodal discourse analysis and combines the textual examination of the basic material with more systematic accounts of inferences and logical reasoning in film and media interpretation (cf. Wildfeuer 2012; Wildfeuer 2013; Wildfeuer 2014a; Bateman/Wildfeuer 2014). Text as the material basis for all analyses plays an important role in this work (see also the introduction to this book) and is the starting point to find out more about the semantic and pragmatic specificities of filmic meaning construction. The pragmatic aspect of these approaches to film analysis is thus located at the foundation of an integrative account for film interpretation that combines various linguistic levels of analysis of both filmic technical features as well as the ‘higher level textual data’ of filmic texts (cf. Bordwell 1989). In the following, we will outline these various textual levels of interpretation in more detail and with the aim of providing a systematic overview of how a pragmatic and textual account of film interpretation may look. This then follows those (rather traditional) accounts in pragmatic film theory that still see these “two fields of force” (the pragmatic field on the one

From Text to Recipient

121

hand and a textually oriented approach on the other, Hesling 1989: 104) as not only opposing ideas, but rather with a “dynamic relation of tension” (Hesling 1989: 104). In contrast to Simon’s formulation of the ‘shift from text to recipient’ (see above), this paper wants to push the notion of text forward into pragmatically based analyses of films with regard to their basic and inherent meaning-making patterns. For this, the definition of the filmic text as given so far will first be discussed in further detail and enriched by a particular model of discourse interpretation, bringing into play a discursive expansion of the filmic text with regard to contextual, social, and cultural knowledge sources as well as recipient-related information needed to fully perform this process of interpretation. The analysis of a short example film excerpt at the heart of this chapter will then focus primarily on the material provided by the filmic text and the pragmatic information needed to construct meaning out of it, before – in a further step – taking into account the role of the recipients and their abductive inferences in the process of interpretation. This will finally result in a more systematic description of all stages and levels involved in the meaning-making process, also including interpretations that go beyond the semantic level of description. We will therefore shortly show at the end of this paper how the provided semantic-pragmatic analysis can support, for example, an approach to the appraisal of film in terms of aesthetic categories, for example. As a consequence, we understand the phrasing of the title of this chapter not in a similar way to Simon’s formulation, but rather as the description of a bridge-building approach between the two units of text and recipient that takes their mutual scope as a starting point for a more detailed examination of their interplay.

2 Filmic Text and Filmic Discourse The aim of providing a textual basis for a semantic-pragmatic analysis of filmic text brings with it the need for a clear and comprehensive definition of film as text, which, as a matter of fact, cannot be taken for granted within all film theoretical approaches and discussions. Although it is a deep-seated convention to talk about films as textual artifacts, a clear delineation of the subject matter from a linguistic, philological, or textological perspective is often not included in theoretical discussions or analytical examinations. In the following, we will first summarize existing and ongoing discussions of the notion of filmic text in film semiotics and the context of multimodal analysis, before going on to provide a distinct definition of how film can be understood as a multimodal text with regard to its semantic and pragmatic analysis. 2.1 Analyses Textuelles and Multimodal Text Definitions Approaches that explicitly deal with a filmic notion of text and textuality are in fact rather rare and can mostly be summarized under the specific context of the analyses textuelles (cf. Blüher/Kessler/Tröhler 1999), a period in the

122

Janina Wildfeuer

1960s and 1970s mainly in French film theory in which the film as languageparadigm was first established. At that time, researchers such as Metz (1966), Barthes (1977), Bellour (1975), and others dealt with questions of semiotically analyzing filmic units and their relationships and thereby determining the status of the filmic artifact as very similar to that of literature, for example. Developments in this context and at that time run almost parallel to a shift of focus in general linguistics to the text as a new object of study. In contrast to the already very forward-looking Metzian definition of a sequence of sounds, images, and music as text (cf. Metz 1974), this focus finally takes into consideration units beyond the written or spoken sentence, although by no means including other than verbal entities. Whereas film text examinations already talk about the dynamic character and the material expansion of the filmic artifact in the 1950s and 1960s, the relatively new discipline of text linguistics at that time still hesitated to expand its scope of description towards all kinds of texts, be they visual/pictorial or acoustic, static or dynamic – a situation that took a few more decades to fully disperse. Nowadays, the situation is different both for film theory as well as within contemporary linguistics. For the former, the acceptability and applicability of the notion of filmic text are widely discussed2 and often critically questioned, sometimes without going into detail about just what text is or can be. The latter, in contrast, has developed a comprehensive framework of multimodal text and discourse analysis that provides theoretical and methodological tools for the detailed examination of filmic (and other media) texts, both from a higher-level interpretative perspective as well as in detailed empirical investigations. Approaches in this context mostly follow basic assumptions of systemic-functional linguistics and social semiotics (cf. Halliday/Matthiessen 2004), including also one of the most fundamental definitions by Halliday/ Hasan (1976: 1–2) that “text is best regarded as a semantic unit; a unit not of form but of meaning.” This understanding of text as a meaning-making entity replaces many definitions concentrating only on verbal texts and their grammatical units (such as sentences, for example) and offers an application to all kinds of communicative artifacts that produce meaning. Following Gunther Kress, one of the founding fathers of the multimodality framework, for example, text is then first and foremost a “social object” (Kress 1993: 221), taking the social dimension in communication as a starting point to think about the understanding and reception of any text. This reception depends on the various semiotic entities and material circumstances that produce limits on the scope of the text: “Texts and their sub-textual entities are realized through various resources; many devices are necessary and available to give shape to text and sub-textual units as well as to the relations between them.” (Kress 2010: 147) As a very general description, this definition is also one of the rather rare, but clear delineations of what is frequently called ‘the multimodal text’.3

From Text to Recipient

123

Other approaches often provide methodological and analytical specifications, but do not discuss the notion of text in detail. Furthermore, ‘text’ is often replaced by ‘discourse’ in this context, bringing into play contextdependency and the dynamics of interpretation within a specific culture or against a particular historical background. Discourse is then often understood as an abstract and overarching entity without any material basis: “The term ‘discourse’ functions in the theory as a resource for constructing epistemological coherence in texts and other semiotic objects. Discourse refers to ‘institutions’ and the knowledge they produce about the world which constitutes their domain […].” (Kress 2010: 110) Both notions of ‘text’ and ‘discourse’ are used similarly and are interchangeable in this context, which often leads to misunderstandings in languages and contexts that maintain a clear distinction between the two terms.4 Similarly, film as an audio-visual artifact is classified as a complex multimodal form, text, or discourse with textual, or discursive, qualities such as coherence and structure (cf. van Leeuwen 1985, 1991; Burn/Parker 2003; Bateman/Schmidt 2012; Wildfeuer 2013; Burn 2014; Wildfeuer 2014a). Researchers using the notion of multimodal filmic text underline that this is not a new feature in film theory, but that it has already been described by early theorists such as Eisenstein or Bazin, who theorized and examined relationships between entities in a filmic text and thus built theories of multimodality without using the term (cf. Burn 2014). In recent years and with the development of theoretical and methodological differentiations of the notion of ‘mode’, for example, and the detailed analysis of the interplay of the various resources in an artifact, the study of film as a multimodal artifact has been further specified and refined. There are now considerable achievements for a comprehensive examination of filmic meaning-making. However, a detailed, analytically viable, and precise definition of the filmic text is in most cases not available in these approaches. Especially concerning the empirical validation of theoretical and methodological advancements, the need for such a definition is still given. As Bateman (2016) points out, it is a current issue in applied (and also film) semiotics to “establish analytic guidelines that encourage even individual text analyses to feed into more general bodies of results and to encourage subsequent empirical probing.” We see the definition of the object of study as a first starting point for a comprehensive description of these guidelines and will, therefore, elaborate on this in the following discussion. 2.2 Towards a Precise Definition of Filmic Text and Discourse In particular with regard to the simultaneous use of both the terms (filmic) ‘text’ and ‘discourse’ within the paradigm of multimodal analysis as well as with regard to the various possible understandings of the two notions

124

Janina Wildfeuer

in the academic landscape and, moreover, in daily life, we will now provide the  necessary terminological clarification of these terms for the analytical examination below. We will see that the individual description and the subsequent comparison of the two concepts and their understanding in various contexts will show considerable differences, in general. However, the aim of this overview and the following discussion is to provide a fruitful combination of both understandings by showing intersections and points of references. For this, we take a first approach to the definition of the filmic text that has already been provided in our earlier works: “[…] we propose a concrete definition of film as a multimodal text which is meaningfully structured by a variety of semiotic modes. It is a dynamic but formally confined artifact in chronological, linear order. It may have intertextual references to further text types and may produce various communicative intentions according to the context.” (Wildfeuer 2014a: 10; see also Wildfeuer 2013) The filmic text according to this definition is seen as a complex system of signs and semiotic resources with a communicative function and is thus understood very similarly to the general definition of multimodal texts provided by Gunther Kress, for example (see above). Furthermore, the definition provides a detailed description of basic qualities of the filmic text and thus directly opens up the field of examination, which then could, for example, focus on the question of structure or intertextuality, or ask for specific communicative acts. In this regard, the definition describes filmic text first and foremost as a single artifact and concrete material object that shares with the traditional linguistic concept of the verbal text the characteristics of being a semantic unit, i.e. carrying meaning, which may, in the case of film, be constructed out of verbal and non-verbal semiotic resources. In particular the latter then demand a certain broadening of the levels of examination due to the fact that the meaning of images or music, for example, or their role in the intersemiotic interplay, is never as explicit as that of verbal units and therefore calls for a more context-dependent interpretation on the basis of general world knowledge and further information sources. It is exactly these features that bring into play the notion and level of discourse. While ‘text’ is in most approaches seen as a concrete “object in its own right” (Halliday/ Matthiessen 2004: 3), ‘discourse’ brings with it the notions of abstractness and context-dependency that do not hold for a single, individual text as a material entity. Instead, meaning construction of multimodal artifacts is always an interpretation in context that takes place on the basis of knowledge sources and socio-cultural background information, i.e. discursive and pragmatic concepts of text interpretation. As a consequence, multimodal discourse analysis deals with both single text examples as well as broader

From Text to Recipient

125

discourses and corpora and asks, for example, for socially constructed knowledge and practices in texts and their contexts and discourse(s) (cf. Kress/van Leeuwen 2001: 4; Bucher 2007: 41). For the following analysis and to firstly and accurately distinguish ‘filmic text’ and ‘filmic discourse’, we thus take the film’s materiality as a crucial differentiating feature. This feature mainly refers to criteria inherent to the text, such as the multitude of semiotic resources and their interrelationships as well as cohesion/coherence and structure of the text. They form the semantic basis for every further interpretation of the filmic text. This focus on the materiality also corresponds to the problem and paradox of the so-called “unattainable text”, as described by Bellour (1975). On the one hand, Bellour defines “the text of the film [as] unattainable because it is an unquotable text” (Bellour 1975: 20): its different matters of expression do not allow characterizing or reproducing it in written form, for example, as is the case for verbal text. Therefore, according to Bellour, its materiality cannot be grasped. On the other hand, he underlines that “the material possession of the work alone permits one full access to the textual fiction since it alone allows one a full experience of the multiplicity of operations carried out in the work and makes it precisely into a text” (Bellour 1975: 20). It is precisely this material possession of the work which is the aim of any textual and semiotic description, focusing on the various semiotic resources available in the text and producing meaning in their interplay.5 With the aim of finding out more about this filmic meaning construction in general and more specific knowledge or socio-cultural structures, this semantic basis can then be seen as the starting point for filmic text and discourse analysis. Consequently, the filmic text is a concrete material artifact that is subordinate to the filmic discourse as an abstract entity and for which recourse to the actual material of the respective text is again necessary. In this sense, any individual occurrence of filmic text is autonomous with its surrounding discourse in that it makes the discourse available in its concrete materializations – either as a singular occurrence or in a serial compound of several occurrences. Following approaches in multimodal discourse analysis, which see meaning construction in multimodal texts and discourses on several strata (cf. Halliday 1978)6, we provide the diagram in Figure 6.1 as an illustration of the several levels of filmic text and discourse. In this stratificational view, the filmic text is the material unit (in gray) whose individual data and semiotic resources construct meaning due to their specific materiality. The filmic data, therefore, constitute the basic level within this hierarchy of film interpretation and represent the elementary substance that is arranged in diverse ways. In multimodal discourse analysis, this arrangement is often described with their respective (ideational, interpersonal or textual) metafunction in the specific context (cf. Halliday 1978). These functions and meanings are influenced by the general context as well as different knowledge sources and the social activity of the recipients in which the text is embedded. This context can be seen as the overall

126

Janina Wildfeuer

DISCOURSE context: knowledge sources social activity

propositional content

TEXT filmic data

ideational

textual

interpersonal

Figure 6.1 The levels of filmic text and discourse (following a more general description of text and discourse in Wildfeuer forthcoming).

environment in which social communication takes place and which delivers regularities for the use of specific semiotic resources. Discourse is represented as realized on all these levels of description, as a superordinate entity which includes the text and cannot be realized without it. It is exactly this principle of realization that arranges the relationship between text and discourse across the strata. Martin/Rose (2003: 5), for example, describe this principle as “a kind of re-coding” that is associated with the concepts of ‘symbolizing’, ‘encoding’, ‘expressing’ and ‘manifesting’. Meanings (in verbal discourse, originally) are therefore always realized across strata, across both levels of discourse and text, and are made manifest in the concrete unit of the text. This is, in particular, the case for film interpretation since the concrete interplay of various resources in filmic text as well as their different functions within the meaning-making processes can only be interpreted by referring to the actual context and on the basis of several knowledge sources concerning the story’s content and characters, for example. The stratificational organization given in Figure 6.1 thus represents a direct alliance between the levels of filmic text and filmic discourse, which can be described in terms of the notion of instantiation as introduced in the systemic-functional linguistics context (cf. Halliday/Matthiessen

From Text to Recipient

127

2004: chap. 1.3.4.).7 Any material unit of text instantiates the abstract unit of discourse in which it is embedded and to which it always has a reference. Furthermore, with the classification above we provide a process of pre-structuring film interpretation both on a semantic as well as pragmatic level of description. Constructing a narrative filmic text via inferences, which is the focus of description in pragmatically focused approaches to film analysis, is a mostly data-driven operation on the basis of the viewer’s perceptual input during the reception of the film. The perceived filmic data is then processed within the different strata and levels of film interpretation to systematically build meaning structures out of the variety of modalities and their intersemiotic composition. Perception and comprehension of the filmic text, that is, at the lower levels of description, thereby build the basis for all bottom-up activity by the recipient, as described by Bordwell (2008: 47): “In comprehending a film, we construe the outputs of filmic perception as representing a hierarchical pattern of actions, a conception, or simply a train of sensuous elements (as in an abstract film). The viewer applies a wide range of knowledge to make sense of film, segment by segment or as a whole, and to give it some literal meaning. Narrative comprehension is the clearest instance.” Since film interpretation is, however, a constant interaction between bottom-up and top-down processes, meaning construction is invariably enhanced by concept-driven processes from above, i.e. from the discourse level, that takes into account the social functions and uses of the textual, i.e. pragmatic, structures. As a consequence, the dynamic interaction between the operations allows a comprehensive overview of film interpretation from both the systematic overview of constructing film narration in the text and the recipient’s interpretation within the social context. The depiction in Figure  6.1 is therefore also designed as a representation of exactly the bridge-building approach we take here in order to combine the view of the filmic text with the view of the recipient’s activity. We will strengthen this view in the following by applying this theoretical conceptualization to the analysis of concrete filmic material, both on the level of text as well as discourse to show the different analytical levels of a comprehensive discourse-pragmatic interpretation of filmic text in operation.

3 Analytical Steps of a Discourse-Pragmatic Interpretation For an illustration of our theoretical implementations given above, we will now provide a multi-level analysis of a filmic extract concerning the film’s materiality and its semantic and pragmatic structures as well as the inference processes, knowledge sources, and contextual information needed for interpretation by the recipient. We will work through the analysis by following both bottom-up and top-down processes through the different strata

128

Janina Wildfeuer

of text and discourse as described in Figure 6.1. For this, we will refer to several analytical frameworks that have been developed within multimodal film analysis in the last few years, focusing on aspects of textuality and discourse (cf. Bateman/Schmidt 2012; Bateman 2013; Wildfeuer 2014a, b). As the analytical example to be discussed in detail in the following, we will focus once again on the famous opening sequence of the film Gravity (Cuarón 2013; see also the analysis of Huvenne in this volume). The very beginning of this scene consists of a black screen with verbal language inserts as well as the title sequence, all displayed in white letters and accompanied by a specific sound atmosphere (see Figure 6.2 and Huvenne’s analysis of sound in this volume). Connected to this short sequence is the following long sequence by an apparently hard cut (see Figure 6.2). In contrast to Huvenne’s precise analysis of the sound space in Gravity, we start our examination with a particular focus on the film’s various semiotic resources and their interplay, i.e. the whole material dimension of the excerpt at the beginning of the film. A first result will then be a clear description of the information available from the text itself. With the aim of describing both the semantic and pragmatic structures available in this sequence, we will then demonstrate how these structures guide the recipient through his/her interpretation on both a textual as well as discursive level. For this, we will first collect available information sources that can be inferred from the film’s material and then discuss possible ways of providing interpretations.

Figure 6.2 Inserts, title sequence, and cut to long shot in Gravity. The inserts in the first shot read “At 600km above the planet Earth the temperature fluctuates between +258 and −148 degrees Fahrenheit. There is nothing to carry sound. No air pressure. No oxygen. Life in space is impossible.”

3.1 Basic Patterns of Meaning-Construction in Gravity With regard to the actual material, the first two shots (as given in Figure 6.2) contain white color on a black background, formed in the shape of written language as verbal inserts and accompanied by sound that already starts together with logos and credits of the film’s production company at the very beginning of the film. In their intersemiotic interplay, these material sources can be seen as semiotic resources, i.e. “semiotically charged organizations of material that can be employed for sign-construction” (cf. Bateman 2011: 20). These resources work, for example, on the level of written language which, as so-called inserts, gives explicit information about the situation in space as well as the film’s title and can, for example, be analyzed with

From Text to Recipient

129

discourse analytical frameworks outlining the propositional content of the sentences (cf., e.g., Kamp/Reyle 1993; Asher/Lascarides 2003). Other parts of the visual level do not seem to deliver similarly clear or explicit, semantic details. Whereas the verbal text presents a description of the situation in space, the black background does not carry a clear semantic content that could similarly be described by a proposition. As a direct visual realization of the described content on the verbal level, it first and foremost creates cohesion with the phrase ‘in space’ as well as coherence between the fade-ins of the verbal inserts and the title sequence and therefore works as an interface between the semantic and pragmatic level of description. The auditory level surrounds the black screen with a forceful, bass-heavy sound atmosphere with increasing and intrusive volume, and thus adds to the overall meaning construction, but does not deliver further explicit propositional content. It is then abruptly stopped and replaced by silence when the black screen and title sequence cuts to the actual space setting. The dynamic visual representation of this setting gives a particularly framed view of the Earth as seen from space. Although most have never been in this situation themselves, recipients can infer this description because of now commonplace general knowledge concerning how the Earth looks when seen from this perspective. The basic textual understanding of this first introduction to the film is thus relatively straightforward and easy to process – provided that the recipient is able to read and understand the semantic content of the verbal inserts. This understanding is always accompanied by background and discourse knowledge brought to the material by the recipient through his/her interest in the film. It can, for example, normally be assumed that spectators are to some extent informed about the content of the film before watching it – due to their knowledge of trailers, film posters, and further summaries and reviews available in the media. This knowledge is processed from a topdown perspective on the level of discourse (see Figure 6.1), since it includes information that is not directly coming from the film’s material, its textual basis. It is, however, directly related to this level, and keywords such as ‘space’ and the film’s title activate associations and hypotheses about the unfolding content from these information sources. 3.2 Discourse Semantics and Pragmatic Structures These associations and hypotheses about the coherence of the shots underlie meaning-making principles that go beyond the simple reading and processing of information available in the text or its surrounding discourse context. Moreover, coherence and structure as textual qualities of the film guide recipients in making logical conclusions about relationships between the various entities and their functional embedding. This is the discourse perspective on film analysis, bringing into play the level of discourse semantics (cf. Bateman 2013):

130

Janina Wildfeuer “The discourse semantics provides defeasible rules of interpretation that are typically applied to show how interpretations can be uncovered dynamically while at the same time also sensibly constraining the range of interpretations that are relevant. In short, discourse semantic rules control when and how world knowledge is considered in the interpretation process […].” (Bateman 2013: 22)

As demonstrated in further detail by Bateman (2013; 2014), every semiotic mode draws upon its very own discourse semantics that delivers a generalization of guiding schemes for contextual interpretation across different contexts (cf. Bateman 2013: 21). The discourse semantics of the semiotic mode of film, therefore, brings with it certain principles and patterns that allow the reconstruction of how the various entities are related to each other and intersemiotically interplay. Whereas the film’s material provides the various units and semiotic resources, we have described above, the level of discourse semantics then delivers possible interpretations for their coherence and structure. A fruitful approach to describe these interpretations is to analyze discourse relations as abstract units holding between segments of a film (cf. Wildfeuer 2014a). By describing the recipients’ inference processes of making sense of the various units and searching for meaningful relations between them, the examination of these film discourse relations then outlines causal and text-structuring circumstances within the film’s story and at the same time provides exactly the defeasible mechanisms and rules that guide the recipient to these interpretations in the specific context and situated discourse. Over the last few years, we have developed a number of approaches dealing specifically with this notion of film discourse relations (cf. Bateman 2007; Bateman/Schmidt 2012; Wildfeuer 2014a, b) and providing distinct sets of relations and their defeasible rules for interpretation so that we are now able to turn back to these theoretical and methodological approaches to the analysis of the example discussed here. For a more detailed description of these approaches, see Bateman (2007), Bateman/Schmidt (2012), Wildfeuer (2014a: chap. 2 & 3) and Wildfeuer (2014b). In the example sequence under discussion, the composition of the film’s beginning includes a title sequence and inserts indicating the setting of the film’s narrative. These inserts introduce the general setting of the film, the space, 600 kilometers above the planet Earth. Whereas the verbal inserts represent the first description of this setting, the moving image of the view of the Earth from space then realizes this description on the visual level. With regard to the interplay of these two semiotic resources, which is, in the context of multimodal discourse analysis, also often described in terms of text-image relations (cf. van Leeuwen 1996; Martinec/Salway 2005; Liu/O’Halloran 2009), this visualization would then give further detail information, a

From Text to Recipient

131

specification or a so-called Elaboration of what is said on the verbal level in the shot before. However, more resources are working together in this sequence than just a simple cooperation between the visual and the verbal level, and it is also in particular the sound which plays an important role for the whole composition. As Huvenne (this volume) points out, “sound space and visual space are not presented in the same way” in this film and sound rather creates its auditory space. This is already the case for the beginning sequence in which the inserts, on the one hand, talk about the impossibility of carrying sound and the absence of air pressure. However, the auditory level, on the other hand, provides a substantial sound atmosphere with increasing volume that abruptly stops. When the film cuts to the space setting, this sound atmosphere is indeed replaced by silence – representing a sharp contrast to the ambiance previously. However, silence on the auditory level represents what has been expressed on the verbal level before: ‘there is nothing to carry sound’. Elaboration on the visual level is thus also supported by the auditory level and, for this short moment, sound space and visual space cohere. The situation then indeed changes only a few seconds later, when new sounds become audible, first only in the background and later with increasing volume, while the camera slowly moves and, finally, a small white object becomes visible in the middle of the frame (see Figure 6.3). As Huvenne describes: “Disconnected from the visual source, we listen to voices and hear radio signals introducing the presence of human beings in an auditory way. The voices approach, words become understandable, and at a certain moment we really identify a woman’s voice and breathing.” (Huvenne this volume) These voices on the auditory level are opposing the assumption made by the verbal inserts and thus represent a contrast to the dominating silence before. This contrast happens due to a dynamic change in the filmic shot, both on the visual as well as auditory level, which has to be interpreted as new information brought to the discourse, i.e. a certain update of existing information, and which has to be set into relation to the preceding discourse. As a consequence, the hypothesis about the Elaboration relation holding between the first introductory sequence and the visual representation in the second shot has to be updated. The inference of a Contrast relation then overrides Elaboration, since the dialogue on the auditory source can be interpreted as semantically dissimilar to the content expressed by the verbal inserts. In our approaches, we provide formal description tools for the demonstration of this inference process to make the defeasible rules holding between the entities more prominent. In the following, we will give only one example for

132

Janina Wildfeuer

Figure 6.3 Two frames from the very long shot at the beginning of Gravity, showing a white object becoming visible.

the description of the inference of Contrast, as discussed above and formally displayed in the following formula:

(

)

(

)

(

(? eπ 1 , eπ 2 , λ ∧ semantic dissimilarity eπ 1 , eπ 2 ) > Contrast eπ 1 , eπ 2 , λ

(

)

)

Here, ? eπ1 , eπ 2 , λ stands for an underspecified relation holding between the two segments eπ 1 and eπ 2 in the specific context λ. eπ 1 and eπ 2 stand for the result of inferring the semantic content from the first part of the sequence, as given in the verbal inserts, as well as from the second part, the visual representation of the space setting along with the dialogue on the auditory level, which can then be generalized as a conversation in space. Under the condition that semantic dissimilarity holds between these two segments, this underspecified relation can then, in the specific context λ, and on the basis of the contextual information described above, be interpreted as Contrast holding between eπ 1 and eπ 2 . The process of dynamically inferring these relations between the entities on the basis of contextual information is exactly the pragmatic task to be performed by the recipients by means of their ability to logically conclude coherence between the segments in the film’s unfolding narrative. Based on the understanding of the semantic content in the first sequence and the perception of the visual and auditory input in the second sequence, that is the material substrate, it is then also necessary to bring into play background knowledge about the technical possibilities for astronauts to speak to each other while working in space. That this is possible helps one understand that the situation shown in the shot is not unreal, but that recipients can actually hear the sounds from the perspective of one of the characters, i.e. due to focalization, for example (see further details in Huvenne’s paper). Knowledge about the actual possibilities in space as well as knowledge about how film is able to represent these possibilities is brought into the interpretation top-down, induced by processes of bottom-up meaning construction (see Figure 6.1 again). It is exactly the specific discourse semantics of the filmic discourse that provides, on the one hand, the ability to construct meaning textually and cross-modally, i.e. with an interplay of various semiotic resources, and which works, on the other hand, with a clear set of discourse relations and their formal rules (as described in Wildfeuer 2014a and by the example above)

From Text to Recipient

133

helping to analyze this interplay in further detail. Furthermore, the discourse semantics makes available interrelations between these two levels and thus mediates between them from both a semantic as well as pragmatic perspective. Consequently, it illustrates that the production of meaning is not – as stated by Odin, for example – entirely based on external determinations, but that the recipients’ inferential activities are dynamically interwoven with information inherent to the text. 3.3 Going Beyond the Semantic and Pragmatic Level This detailed description of the interplay of text and discourse as well as of the semantic and pragmatic perspective not only supports a more comprehensive examination of how filmic meaning is constructed, but also allows a discussion of this semantic and pragmatic basis with regard to higher-level interpretations brought to the film from a broader media studies or philosophical approach. We will again work with only one example of these interpretations to demonstrate interrelations between their results and those examined with our approach to film analysis. Chung (2015a), for instance, analyses film’s emotional and affective reception in terms of certain aesthetic strategies and shows that the specific composition at the sound and visual levels in Gravity includes specific strategies for overwhelming recipients during their reception of the film, which she relates to the category of the so-called filmic ‘sublime’, following the Kantian aesthetic and philosophical concept. By referring to Kant’s “Analytic of the Sublime” in his “Critique of Judgment” (1914 [1790]), in which he describes these sensory, overwhelming experiences by notions of the mathematically and the dynamically sublime, Chung works out that recipients are confronted with a world whose understanding and interpretation is often going beyond usual accomplishments of our senses. The scene discussed above, for example, supports this strategy of overwhelming by the radical contrast constructed between the two sensory channels on the one hand, as well as the replacement of the space setting by a focus on the characters and thus a micro-perspective contrasting with the over-dimensional wideness (cf. Chung 2015a), on the other. Chung explains that it is exactly the precarious size ratio which attributes a sensory quality to the emptiness of space and its spatio-temporal dimension of immensity. In the beginning, the silence of the space setting and the breadth and largeness of the Earth shown in the shot not only address and challenge the viewers’ perceptiveness and captivate their attention, but also make it possible for them physically to experience the dynamic orbital motion around the earth. Following the theory of the filmic sublime developed by Chung, this filmic experience can then be described as a case of the mathematically sublime (cf. Chung 2015a: 115), which is furthermore supported by the dynamically unfolding discourse and its changes of focus to the dialogue and the micro-perspective on the characters. It is precisely this contrasting

134

Janina Wildfeuer

composition of the film’s audio-visual structure which we have now analyzed with the help of the Contrast relation on the discourse semantics level and which allows, as a result of the film’s textual qualities, a manifestation of the philosophically based interpretation. Chung analyzes various other scenes and structures in Gravity that provide further evidence of experiences of the filmic sublime in terms of the mathematically and dynamically sublime as well as with regard to empathy with certain characters (see also Chung 2015b). In most of these instances, the sound level plays a significant role, and Huvenne’s more detailed analysis needs to be taken into consideration. While Chung herself is pursuing in her work an examination of the textual qualities from a general, film analytical perspective, our detailed semantic-pragmatic approach on both the textual as well as discursive basis adds to her analysis a focus on the recipient’s semantic and pragmatic work during the interpretation process as well as on the various knowledge sources involved in this process. In general, all instances mentioned by Chung can similarly be analyzed by the approach demonstrated above or with regard to further film analytical tools developed from the textual and multimodal perspective, such as Tseng’s approach of cohesion and character tracking (cf. Tseng 2013). The comprehensive analysis of combining bottom-up and top-down analyses as well as lower level and high-level interpretations represents a further step towards bridging the so-called semantic gap holding between these levels, an issue that is now more and more addressed in interdisciplinary approaches in the humanities (cf., among others, Dorai/Venkatesh 2003). In work on film experience, for example, Suckfüll (2007) and many others underline that it is still a challenge for empirical research to isolate, on the one hand, stimuli within a piece of art and to clearly identify, on the other hand, the respective emotional or affective reaction concerning the overall context of the film’s diegesis. This aim can be further supported by the textual analysis provided above, i.e. with both a semantic as well as pragmatic approach, in that textual clues are made explicit and, in a further step, can be related to interpretations beyond the semantic level of description, such as, for example, the appraisal of the film in terms of the filmic sublime, as addressed by Chung.

4 Expanding the Toolbox of Film Analysis – From Semantic Descriptions to Higher-Level Interpretation We have shown in this chapter what a contemporary approach to pragmatic film analysis on the basis of developments in more recent film semiotic and multimodal discourse analytical accounts can look like. As demonstrated by the example analysis above and with regard to further examinations on a stronger empirical basis, it has become clear that a detailed description

From Text to Recipient

135

of the film’s materiality on the one hand, and the necessary interpretation steps in terms of abductive inferences on the other, can shed further light on filmic textual qualities and their ability to guide and constrain the recipient’s interpretation. With a focus on the textual and discursive as well as semantic and pragmatic levels of description, we have highlighted that contextual details brought to the text by the recipient have to be taken into consideration. Furthermore, by formally describing how these contextual details are part of the inference processes (as shown in the example rule of inference used in section 3.2), we have described them as being part of the textual-discursive interpretation of the film as a multimodal artifact. This goes beyond the traditional film theoretical view of pragmatic analysis that sees film interpretation as a fully external process of meaning making (see above). By directly relating knowledge sources and contextual details to information available in the text itself, i.e. in its materiality and semiotic composition, the interrelation between technical features of the film and the recipients’ interpretation can be made stronger and more reliable. As we have furthermore seen in section 3, there is always reciprocal dependency and mutual influence between the various textual and discursive strata, as shown in Figure 6.1 in section 2. Nevertheless, the provided definitions of text and discourse help distinguish significant levels of description within a comprehensive film analytical approach that is able to take into consideration both bottom-up and top-down processes. For this, the textual level can mainly focus on the film’s material basis regarding the various semiotic resources and their semantic potential, whereas the discursive level then brings into play the respective knowledge and information sources needed to construct meaning out of these resources. Concerning broader empirical examinations, the filmic text is always the single and concrete material object that might be embedded in a specific, thematically focused context and discourse. This comprehensive, semantic-pragmatic approach can then not only ask how film interpretation, in general, takes place but also which interpretive possibilities are available from the inherent, significant patterns of the text. These patterns shed light on interpretation which goes beyond the pure semantic meaning of the text and contributes to questions such as those raised in the other contributions to this book and beyond: questions in literary analysis, cultural studies, or cognitive film theory, for example.

Notes 1. Buckland admits, however, that not all pragmatic theories go this far (cf. Buckland 2000: 80–81). Nevertheless, the critique and rejection of the textual analysis of films could not be more topical today, summarized for example in a recent talk (in January 2016) in the Kracauer lectures at the Goethe University in Frankfurt am Main, Germany, by Richard Dyer on “The Persistence of Textual

136

2. 3.

4.

5.

6.

7.

Janina Wildfeuer Analysis” (see also the introduction to this book). Dyer’s conclusion, however, is not to stop doing textual analysis, because “there is not such thing as not doing textual analysis” (cf. Dyer 2016). See, for example, Pauleit (2009), Bateman/Kepser/Kuhn (2013), Bateman (2013) that all explicitly deal with the question of textuality, often (but not always) coming from a linguistic background. Interestingly, even glossaries with key terms and topics of multimodal research do not include a definition for (multimodal) ‘text’ or discourse (see, for example, http://www.multimodalityglossary.wordpress.com or Norris/Maier 2014). It is again mainly the German-speaking context and the developments in text linguistics and discourse analysis in German philology that are faced with these terminological problems, often seeing ‘text’ as a static, verbal entity only and replacing textual analysis with broader discourse analytical examinations (cf. Warnke/Spitzmüller 2011; Heinemann 2011; Wildfeuer forthcoming). Furthermore, the Foucauldian notion of discourse as a very dominant concept in the broad field of humanities is the reference point of many discussions about the term ‘discourse’, but to a large extent does not correspond with that of the systemic-functional context. Without clear determinations and clarifications, this then also often leads to disagreements. It is of course also much easier nowadays to ‘quote’ filmic text with the help of video production tools and further digital analytical instruments. Nevertheless, the written form of summarizing and describing film interpretation is still the most frequent style of analysis. The notion of stratification goes back to the basic distinction between content and expression plane according to Hjelmslev (1961) and has been further developed and applied to multimodal artifacts within approaches to systemic-functional linguistics (cf. Halliday 1978). The idea is to further divide Hjelmslev’s levels into phonology and graphology on the one hand (expression plane) and lexicogrammar and semantics on the other hand (content plane). Martin (1992) then describes the level of semantics as a specific discourse semantics (see below). See for further details the discussion by Halliday/Matthiessen for verbal texts (2004: 27): “The system of a language is ‘instantiated’ in the form of text. A text may be a trivial service encounter, like ordering a coffee, or it may be a momentous event in human history, like Nelson Mandela’s inaugural speech; in either case, and whatever its intrinsic value, it is an instance of an underlying system, and has no meaningful existence except as such. A text in English has no semiotic standing other than by reference to the system of English (which is why it has no meaning for you if you do not know the language).”

Bibliography Asher, N./Lascarides, A. (2003). Logics of Conversation. Cambridge: Cambridge University Press. Barthes, R. (1977). Image – Music – Text. London: Fontana Press. Bateman, J.A. (2007). “Towards A Grande Paradigmatique of Film: Christian Metz reloaded”. Semiotica 167, 1(4), pp. 13–64.

From Text to Recipient

137

Bateman, J.A. (2011). “The Decomposability of Semiotic Modes.” In: O’Halloran, K./Smith, B. (eds.). Multimodal Studies: Multiple Approaches and Domains. London/New York: Routledge, pp. 17–38. Bateman, J.A. (2013). “Filmische Textualität jenseits der narrativen Instanz.” In: Bateman, J.A./Kepser, M./Kuhn, M. (eds.): Film_Text_Kultur: Beiträge zur Textualität des Films. Marburg: Schüren, pp. 88–115. Bateman, J.A. (2014).: Text and Image. A Critical Introduction to the Verbal/Visual Divide. London/New York: Routledge. Bateman, J.A. (2016). “Methodological and Theoretical Issues for the Empirical Investigation of Multimodality”. In: Klug, N.-M./Stöckl, H. (eds.). Sprache im multimodalen Kontext. Language and Multimodality. Berlin: de Gruyter Mouton (forthcoming). Bateman, J.A./Schmidt, K.-H. (2012). Multimodal Film Analysis. How Films Mean. London/New York: Routledge. Bateman, J.A./Kepser, M./Kuhn, M. (2013). Film_Text_Kultur. Beiträge zur Textua-lität des Films. Marburg: Schüren. Bateman, J.A./Wildfeuer, J. (2014). “A Multimodal Discourse Theory of Visual Narrative”. Journal of Pragmatics 74, pp. 180–218. Bellour, R. (1975). “The Unattainable Text”. Screen 16(3), pp. 19–27. Björkvall, A. (2012). “Multimodality”. In: Östmann, J.-O. /Verschueren, J. (eds.). Handbook of Pragmatics. Amsterdam: John Benjamins, pp. 1–20. Blüher, D./Kessler, F./Tröhler, M. (2010). “Film als Text: Theorie und Praxis der analyse textuelle”. montage AV 8(1), pp. 3–7. Bordwell, D. (1989). Making Meaning. Inference and Rhetoric in the Interpretation of Cinema. Cambridge: Harvard University Press. Bordwell, D. (2008). Poetics of Cinema. New York: Taylor & Francis. Bredekamp, H. (2010). Theorie des Bildakts. Frankfurt am Main: Suhrkamp. Bucher, H.-J. (2007). “Textdesign und Multimodalität. Zur Semantik und Pragmatik medialer Gestaltungsformen”. In: Roth, K.-S./Spitzmüller, J. (eds.). Textdesign und Textwirkung in der massenmedialen Kommunikation. Köln: UVK, pp. 49–76. Buckland, W. (1995). The Film Spectator. From Sign to Mind. Amsterdam: Amsterdam University Press. Buckland, W. (2000). The Cognitive Semiotics of Film. Cambridge: Cambridge University Press. Burn, A. (2014): “The Kineikonic Mode: Towards a Multimodal Approach to Moving-Image Media”. In: Jewitt, C. (ed.). The Routledge Handbook of Multimodal Analysis. London/New York: Routledge, pp. 375–385. Burn, A./Parker, D. (2003). Analysing Media Texts. London: Continuum. Casetti, F. (1983a). “Looking for the Spectator”. Iris 2, pp. 15–29. Casetti, F. (1983b). “Les Yeux Dans Les Yeux”. Communications 38, pp. 78–97. Chung, J. (2015a). Das Erhabene im Kinofilm. Ästhetik eines gemischten Gefühls. Marburg: Schüren. Chung, J. (2015b). “GRAVITY (2013) – Ein Paradebeispiel für die Ästhetik des Erhabenen im gegenwärtigen Kino”. In: Preußer, H.-P. (ed.). Sinn und Sinnlichkeit im Kino. Zur Interdependenz von Körperlichkeit und Textualität in der Filmrezeption. Marburg: Schüren, pp. 103–128. Dagrada, E. (1987). “The Diegetic Look. Pragmatics of the Point-of-View Shot”. Iris 7, pp. 111–124.

138

Janina Wildfeuer

Desilla, L (2011). “Implicatures in Film: Construal and Functions in Bridget Jones Romantic Comedies”. Journal of Pragmatics 44, pp. 30–53. Dorai, C./Venkatesh, S. (2003). “Bridging the Semantic Gap with Computational Media Aesthetics”. IEEE Multimedia, pp. 15–17. Dyer, Richard (2016). “The Persistence of Textual Analysis”. Talk given as part of the Krakauer Lectures in Film and Media Theory at Goethe-University Frankfurt, Germany, 26 January 2016. Video Online: http://www.kracauer-lectures.de/de/ winter-2015–2016/richard-dyer/ [last accessed: 01 March 2016]. Elsaesser, T. (1995). “From Sign to Ming. A General Introduction”. In: Buckland, W. (ed.). The Film Spectator: From Sign to Mind. Amsterdam: Amsterdam University Press, pp. 9–17. Halliday, M.A.K. (1978). Language as Social Semiotic: The Social Interpretation of Language and Meaning. London: Arnold. Halliday, M.A.K./Matthiessen, C.M.I.M. (2004). An Introduction to Functional Grammar. 3rd Edition. London: Arnold. Halliday, M.A. K./Hasan, R. (1976). Cohesion in English. London: Longman. Heinemann, W. (2011). “Diskursanalyse in der Kontroverse”. tekst y diskurs – text und diskurs 4, pp. 31–67. Hesling, W. (1989). “Documentary Film and Rhetorical Analysis”. In: de Greef, W./ Hesling, W. (eds.). Image – Reality – Spectator. Essays on Documentary Film and Television. Leuven/Amersfoort: Acco, 101–131. Hjlemslev, L. (1961). Prolegomena to a Theory of Language. Madison: University of Wisconsin Press. Huvenne, M. (this volume). “Editing Spaces as an Audio-Visual Composition” (see chapter 3). Janney, R.W. (2012). “Pragmatics and Cinematic Discourse”. Lodz Papers in Pragmatics 8(1), pp. 85–114. Kamp, H./Reyle, U. (1993). From Discourse to Logic. Introduction to Modeltheoretic Semantics of Natural Language, Formal Logic and Discourse Representation Theory. Doordrecht: Kluwer Academic Press. Kant, Immanuel (1914 [1790]). Critique of Judgement. Translated with Introduction and Notes by J.H. Bernard. Second Edition, Revised. London: Macmillan & Co. Kress, G. (1993). “Against Arbitrariness: The Social Production of the Sign as a Foundational Issue in Critical Discourse Analysis”. Discourse & Society 4(2), pp. 169–193. Kress, G. (2010). Multimodality. A Social-Semiotic Approach to Contemporary Communication. London/New York: Routledge. Kress, G./van Leeuwen, T. (2001). Multimodal Discourse. The Modes and Media of Contemporary Communication. London: Arnold. Kobow, B.S. (2007). See What I Mean. Understanding Film as Communicative Actions. Paderborn: mentis. Liu, Y./O’Halloran, K. (2009). “Intersemiotic Texture: Analyzing Cohesive Devices Between Language and Images”. Social Semiotics 19(4), pp. 367–388. Martin, J.R. (1992). English Text – System and Structure. Amsterdam: John Benjamins. Martin, J.R./Rose, D. (2003). Working with Discourse: Meaning beyond the Clause. London: Continum. Martinec, R./Salway, A. (2005). “A System for Image-Text Relations in New (and Old) Media”. Visual Communication 4(3), pp. 337–371.

From Text to Recipient

139

Metz, C. (1966). “La Grande Syntagmatique du Narratif”. Communications 8, pp. 120–124. Metz, C. (1974). Film Language: A Semiotics of the Cinema. Chicago: University of Chicago Press. Metz, C. (1991). L’Énonciation Impersonelle, ou Le Site du Film. Paris: Merdiens Kliensieck. Norris, S./Maier, C.D. (2014). Interactions, Images, and Texts. A Reader in Multimodality. Boston/Berlin: de Gruyter. Odin, R. (1985). “Pour une Sémio-Pragmatique du Cinéma”. Iris 1(1), pp. 67–81. Odin, R. (1995). “For A Semio-Pragmatics of Film”. In: Buckland, W. (ed.). The Film Spectator: From Sign to Mind. Amsterdam: Amsterdam University Press, pp. 213–226. Pauleit, W. (2009). “Die Filmanalyse und ihr Gegenstand. Paratextuelle Zugänge zum Film als offenem Diskursfeld”. In: Gwozdz, A. (ed.). Film als Baustelle. Das Kino und seine Paratexte. Marburg: Schüren, pp. 37–57. Peirce, C.S. (1931–1958). Collected Paper of Charles Sanders Peirce. Harvard: The Belknap Press of Harvard University Press. Sachs-Hombach, K. (2011). “Prädikative und modale Bildtheorie”. In: Diekmannshenke, J./Klemm, M./Stöckl, H. (eds.). Bildlinguistik. Theorien – Methoden – Fallbeispiele. Berlin: Erich Schmidt, pp. 97–120. Saussure, F. de (1959 [1915]). Course in General Linguistics. London: Peter Owen Ltd. Simons, J. (1995). ““Enunciation” – From Code to Interpretation”. In: Buckland, W. (ed.). The Film Spectator: From Sign to Mind. Amsterdam: Amsterdam University Press, pp. 192–206. Suckfüll, M. (2007): “Emotionale Modalitäten der Filmrezeption”. In: Bartsch, A./ Eder, J./Fahlenbrach, K. (eds.). Audiovisuelle Emotionen. Emotionsdarstellung und Emotionsvermittlung durch audiovisuelle Medienangebote. Köln: Herbert von Halem, pp. 218–237. Tseng, C. (2013): Cohesion in Film. Tracking Film Elements. Basingstoke: Palgrave Macmillan. van Leeuwen, T. (1985). “Rhythmic Structure of the Film Text”. In: van Dijk, T. (ed.). Discourse and Communication: New Approaches to the Analysis of Mass Media Discourse and Communication. Berlin: de Gruyter, pp. 216–232. van Leeuwen, T. (1991). “Conjunctive Structure in Documentary Film and Television”. Continuum 5(1), pp. 76–114. van Leeuwen, T. (1996). “Moving English. The Visual Language of Film”. In: Goodman, S./Graddol, D. (eds.). Redesigning English – New Texts, New Identities. London: Routledge, pp. 81–105. Warnke, I.H./Spitzmüller, J. (2011). Diskurslinguistik. Eine Einführung in Theorien und Methoden der transtextuellen Sprachanalyse. Berlin: de Gruyter. Wildfeuer, J. (2012). “Intersemiosis in Film. Towards a New Organization of Semiotic Resources in Multimodal Filmic Text”. Multimodal Communication 1(3), pp. 276–304. Wildfeuer, J. (2013). “Der Film als Text. Ein Definitionsversuch aus zeitgenössicher linguistischer Sicht”. In: Bateman, J.A./Kepser, M./Kuhn, M. (eds.). Film_Text_ Kultur. Beiträge zur Textualität des Films. Marburg: Schüren, pp. 32–57. Wildfeuer, J. (2014a). Film Discourse Interpretation. Towards a New Paradigm of Multimodal Film Analysis. London/New York: Routledge.

140

Janina Wildfeuer

Wildfeuer, J. (2014b). “Coherence in Film: Analysing the Logical Form of Multimodal Discourse”. In: Maiorani, A./Christie, C. (eds.). Multimodal Epistemologies. Towards an Integrated Framework. London/New York: Routledge, pp. 260–274. Wildfeuer, J. (forthcoming). “Diskurslinguistik und Text”. In: Warnke, I.H. (ed.). Handbuch Diskurs. Handbücher Sprach- und Kommunikationswissenschaft. Berlin/Boston: de Gruyter (forthcoming).

Filmography Gravity (2013). Alfonso Cuarón. Warner Brothers. UK/USA.

7

Intermediality in Film A Blending-Based Perspective John A. Bateman

1 Introduction: Meaningful Mixtures The notion of mixing materials of diverse types and provenances is increasingly seen to be characteristic of many creative artifacts and performances. Explicitly self-aware combinations and re-use are already sufficiently well established to receive substantial attention in their own right. Such combinations go under a variety of labels, such as bricolage (cf. Derrida 2001[1978]: 360), pastiche (cf. Veale 1997), mashups (cf. Sonvilla-Weiss 2010), ‘remediation’ (cf. Bolter/Grusin 2000; Gunkel 2016), and more. Each of these constructs aligns with a distinct constellation of theoretical and descriptive perspectives. Specifically in the context of film studies, many kinds of mixtures have been described building on the notion of ‘genre’; scholars such as Janet Staiger (2007), Kristin Thompson (e.g., 1999), Stephen Neale (2000) and Rick Altman (1999) have all discussed, particularly for Hollywood film, the role played by more or less deliberate admixtures of diverse generic sources. Recognition of just how widespread this practice is can be seen in the following dialogue fragment from The Player (1992), Robert Altman’s perhaps not-so-satirical take on the creative processes of the Hollywood film industry, with screenplay by Michael Tolkin based on Tolkin’s book of the same name. In this scene, overheard at the beginning of the film, a prospective writer is pitching his idea for a new production to the main character, Griffin Mill: WRITER: Does political scare you? MILL: Political doesn’t scare me. Radical

political scares me. Political political scares me. WRITER: This is politely politically radical MILL: is it funny? WRITER: it’s funny. MILL: It’s a funny political thing. WRITER: And it’s a thriller, too, and it’s all at once. It’s a story about a senator, a bad-guy senator at first. MILL: I see, so so sort of a cynical, political thriller comedy.

142

John A. Bateman Yeah, but it’s got a heart, in the right spot. Anyway, he has an accident. MILL: an accident WRITER: Yeah, and he becomes clairvoyant, like a psychic. MILL: Oh I see; so it’s kind of a psychic, political, thriller comedy with a heart. WRITER: With a heart, and er, not unlike ‘Ghost’ meets ‘Manchurian Candidate’. MILL: Go on, go on, I’m listening. WRITER:

While certainly exaggerated for humorous effect, actual practice is often suggested to be not so far removed from this and it is by no means unusual to find descriptions of films couched precisely in such terms. The mix may even be declared quite openly, as in the titles of Cowboys & Aliens (Favreau  2011) or Pride and Prejudice and Zombies (Steers 2016), evidently functioning as distinctive marketing features in their own right. Although potentially susceptible to critique as repetitive repackaging for commercial gain, such combinations of known elements are by no means necessarily negative (cf. Lefait/Ortoli 2012). Combining the familiar to create novel situations, stories, and artifacts can equally well be seen as an integral part of the creative process. As David Bordwell, for example, observes, again with respect to Hollywood: “Innovations often took the form of genre-blending. The SF family adventure (Close Encounters of the Third Kind), the SF horror film (Alien), the SF war movie (Aliens, 1986), and SF noir (Blade Runner) all opened new niches that sustained variants for decades” (Bordwell 2006: 53) The creativity comes in both the particular selections made and the novelty and productivity of the resulting combinations. Nevertheless, just how such processes of combination operate is still subject to considerable debate and uncertainty. Problems for both technical and theoretical description abound. One commonly used metaphor for discussing combinations particularly of genres is that of ‘hybridity’ (e.g., Bordwell/Thompson 1979; Schweinitz 2002; Liebrand/Steiner 2004; Jaffe 2008). This often appears to describe relatively intuitively what happens when we take, for example, the genre characteristics of a Western and a science-fiction film, or multiple hybrids such as a ‘psychic, political, thriller comedy with a heart’, but more precise characterizations of the phenomenon are rare. One of the principal problems is that it has proved difficult to provide sufficiently clear definitions of just what constitutes any ‘individual’ genre supposedly functioning within a hybrid. Indeed, the evident biological overtones of the metaphor

Intermediality in Film

143

have themselves come under considerable criticism. Staiger, for example, argues that ‘hybrid’ should be kept essentially for cases of cross-cultural ‘mixing’, which is evidently not the case for most Hollywood genre-mixes, which she consequently characterizes as ‘in-bred’. And Allen (2013), following Staiger’s ideological critiques, sets out an even more far reaching rejection of hybridity and its application to genre, arguing instead in favor of Fauconnier/Turner’s (2003) notion of ‘blending’ from cognitive science – thereby picking up further Bordwell’s suggestive phrasing above. In this chapter, we will examine this latter suggestion in more detail, arguing that the concept of blending may in fact offer an analytic tool going well beyond issues of genre and which may prove useful for probing occurrences of creative mixtures of diverse kinds at various levels of granularity. In fact, when considering the emergence of genres and genre mixtures over time, it has already been suggested that a broader view of where such mixtures take their materials and organizational forms will be necessary, even for an adequate account of genre. There is no requirement, for example, that genres and their mixtures should only import sources from within individual media – as Neale observes concerning the formation of new genres in film over time: “What, meanwhile, is particularly striking about this historical sketch is the extent to which many genres either originated in forms and institutions of entertainment other than the cinema or were (and are) circulated additionally by them.” (Neale 1990: 61) Neale lists as examples the relationships between melodrama and earlier stage conventions, between comedy and vaudeville and the circus as well as the comic strip, between musicals and Broadway shows, and between the Western and earlier mass produced fiction and magazines. This also resonates well with a more semiotic definition of media set out in, for example, Bateman (2016), in which media are seen as socio-culturally and historically anchored melting pots for combinations of semiotic resources as meaning-making practices. It therefore appears likely that a broader ability to pick out what is offered from different media will be important both for film research in general and for more complete accounts of genre mixtures in particular. Blending suggests a powerful mechanism for characterizing just how this may operate and so, as a first step towards applying blending to a broader range of mixing phenomena in film, we will consider the potential role of blending in the rather more small scale ‘mixes’ characteristic of intermedial, or cross-media, references in film. As emphasized by Irina Rajewsky, in this kind of intermediality only one medium is present, that ‘carrying’ (simulating, imitating, depicting) the other(s) – that is, intermedial references are

144

John A. Bateman “understood as meaning-constitutional strategies that contribute to the media product’s overall signification: the media product uses its own media-specific means, either to refer to a specific, individual work produced in another medium […], or to refer to a specific medial subsystem (such as a certain film genre) or to another medium qua system [...].” (Rajewsky 2005: 52–53)

Approaching this increasingly widespread phenomenon from the perspective of blending will require that we articulate more finely some of the relevant properties of distinct media and their organization to show these operating within film. It will be suggested below that this may provide a further point of access for studies of the motivations for, and effects of, employing such media combinations at all, as well as showing how the notion of blending can play a role in formally distinguishing rather different cases of ‘mixing’ in film.

2 Blending in Film and the Role of ‘Structure’ The main focus of this chapter lies in a twin-pronged theoretical exploration of the extent to which a suitably formalized notion of blending might (a) offer a treatment of some aesthetic and design choices taken in film that have not been sufficiently motivated previously and (b) extend the analytic toolkit available for exploring the phenomena of intermediality as such. Applying a notion of blending that goes beyond more impressionistic or informal accounts will prove central to this aim. The formal foundations for the notion of blending assumed rely on the mechanisms set out in Kutz et al. (2015), which in turn draw on and refine Goguen’s (2003) treatment of blends as ‘semiotic morphisms’. The original notion of blending was developed in the cognitive linguistic tradition to characterize the conceptual integration of diverse kinds of information (Fauconnier 1997). Information integration appears to play a crucial role in intelligent behavior and so is clearly a key issue for cognitive models. Blends, in general, are similar to more formal considerations of metaphor (cf. Black 1962) in which a correspondence is established between two (or more) ‘domains’ of information. Two important differences between metaphors and blends in their respective development histories, however, are: (i) metaphors are directed, i.e., information from a specified source domain is transferred to a target domain, whereas blends are not – two (or more) domains are considered equal partners in the creation of a blend; and (ii) as just indicated above, blends have from the outset been strongly related to cognitive concerns. Thus, Fauconnier’s (1997) original notion of blending was expressed in terms of suggestive cognitive ‘mappings’ relating mental domains. These differences have since been largely eroded, however; metaphors are commonly seen as being ‘creative’ and, with the advent of

Intermediality in Film

145

Conceptual Metaphor Theory (CMT), have in any case been extended so as to stand in relation to cognitive concerns (Lakoff/Johnson 1980). Where blending now offers considerable potential for moving ahead is in its degree of precision. Indeed, a general critique of earlier approaches to both metaphor and, subsequently, blending was their somewhat vague formulations. Mappings between domains were commonly sketched in diagrammatic form and then discussed informally, making it difficult to characterize more finely just what any particular blend or metaphor was achieving and how. For this, it is necessary to make the intuitively appealing idea of merging information sufficiently precise so that automatic reasoning and inference are supported; we will illustrate why this is important below. Approaches targeting greater precision have most commonly been developed in computational contexts, since there the focus of attention must be on providing algorithms and techniques that allow the consequences of a metaphor or blend to be ‘calculated’ – thereby modeling the evident productivity of metaphor or blend construction in its own right rather than suggesting this solely in post hoc interpretative descriptions. More formal approaches now include metaphorical reasoning, analogical reasoning, and techniques of ‘structure mapping’ across domains (cf., e.g., Gentner 1983; Veale 1997; Gust et al. 2003; Pereira 2007). Goguen’s framework is to be seen in this tradition, although focusing in greater detail than hitherto on mathematical foundations. Goguen (2003) consequently characterizes formally just what classes of mappings need to be involved if the properties typically desired of blends are to follow. Although, as we shall see in a moment, this is now changing, earlier work on blending focused primarily on linguistic examples. A standard case commonly used for illustration is then the distinction in meaning observable between the words ‘houseboat’ and ‘boathouse’ (Fauconnier 1997; Fauconnier/Turner 2003). This pair of words offers a particularly striking contrast since, on the one hand, they are both clearly related in terms of the meanings they draw on – i.e., the meaning of ‘house’ and the meaning of ‘boat’ – while, on the other hand, their respective compositions generate quite different meanings. Briefly working through this example will show the importance of providing more detail for the kinds of mappings involved. In short, blending shows how these respective meanings emerge by a systematic process of structural composition that determines just how specific facets of the contributing meanings may be picked out for recombination. The difference in meanings then arises because different facets are being picked out in the two cases. This is suggested slightly more formally in the two ‘blending diagrams’ shown in Figure 7.1, adapted from Goguen/Harrell (2010) and several earlier discussions. The overall structure of a blending diagram sets out the basic ingredients required for blending to apply. At least four abstract ‘spaces’ are required for each blend: two ‘input spaces’ (shown in the center of each diagram), a ‘shared’ or ‘generic’ space (shown at the bottom of each diagram), and

146

John A. Bateman

Figure 7.1 The classic blending example: the respective semantics of the words ‘boathouse’ (left) and ‘houseboat’ (right) generated by two contrasting blending diagrams operating over the same ‘input spaces’.

the resulting blended space (shown at the top of each diagram).1 The ‘bottom’ space provides the common ground for comparing and relating the two  spaces in the middle so that a blend can be generated. The arrows between spaces then make explicit just which entities and relations are to be transferred, or ‘mapped’, between the spaces contributing to the blend. In the present example, therefore, each of the two input spaces offers a ‘theory’ of a relevant aspect of meaning: the left hand oval in each diagram is a theory of houses, specifying that a house has a resident and is situated on land, the right hand oval in each diagram is a theory of boats, specifying that a boat may have passengers and is situated on water. The lower generic space picks out a common ground for relating these theories: in this case, there are objects that some person can use in some location. The mappings between the generic space and the input spaces show how the common ground can generate both ‘houses’ and ‘boats’ as specific cases of using some object in some location. Blending then picks specific objects and relations from the two input spaces, preserving the overall structure within each space while changing or combining entities to be placed within the blended structure. In the left hand diagram, the resident living in a house on land is mapped to a ‘boat’ rather than a person, yielding a boathouse; in the right hand diagram, the resident in a house is mapped to a passenger living in/ riding in a boat on water, yielding houseboat. Following the arrows between the spaces is then equivalent to making the changes indicated along those arrows and so precisely characterizes the generation of the blend. Blending at its most abstract is consequently a model of how understanding of certain domains can be ‘transferred’ to offer understandings of others (for example, how an understanding of ‘houses’ and ‘boats’ can be transferred to offer understandings of ‘boathouses’, etc.), thus showing many similarities with more traditional processes such as metaphor and analogical reasoning. In this respect, blending diagrams capture mechanisms that generalize those often assumed for treatments of metaphor.

Intermediality in Film

147

One of the most powerful innovations and mechanisms that the account of blending provides is the notion of running the blend (Fauconnier/Turner 2003: 48) – that is, once a blended space has been created, this can be used for deriving further inferences or expectations concerning the kind of ‘world’ that has been created. This is the main mechanism by which the creative force of a blend makes itself felt and is where further formalizations of inferencing and reasoning become essential: such inferences can then lead to structures emerging that are not present in the original input spaces. The value of this for creative thought has been suggested across many domains: a standard example is the development of models of the atom by analogy to the already existing understanding of the solar system (cf. Coulson 2001). Turner (2014) consequently argues that blending plays a fundamental role for human creativity at large and provided one capability beyond all others that marked the emergence of homo sapiens and sophisticated cultural diversification. Interest in both blending and metaphor as powerful mechanisms for relating distinct domains has now broadened considerably beyond the kinds of linguistic phenomena just illustrated and the application of notions similar to metaphor or blending has been explored across many sensory modalities. Charles Forceville has been particularly active in promoting this direction of research and has demonstrated the beneficial application of metaphor and blending for a variety of visual materials, including advertisements, comics, and film (cf., e.g., Forceville 2009, 2016). Cognitive metaphors have recently also been taken up in more detail for film analysis by, for example, Kappelhoff/ Müller (2011), Kappelhoff/Greifenstein (2016) and Coëgnarts/Kravanja (2012) and so it would be interesting to consider these more from the perspective of blending and the formal mechanisms provided for blending as well. In the context of our current discussion on intermedial references, the simplest cases where some kind of blending might be usefully recognized are where there is some direct citation or reference in film to some configuration or event from another medium that is likely to be known to the audience. Staying with Robert Altman, for example, we can consider the often discussed cross-media quotation of da Vinci’s ‘Last Supper’ in the film Mash (1970), as shown in Figure 7.2. This scene groups the main characters in

Figure 7.2 Intermedial citation of Leonardo da Vinci’s ‘Last Supper’ (shown right) in Robert Altman’s Mash (1970, shown left) together with correspondence ‘mappings’ with the original.

148

John A. Bateman

the film behind a long table to say goodbye to one of their members, who believes (mistakenly) that he is committing suicide. This grouping gains an extra visual appeal by having the characters take up the relative positioning and gestures exhibited in da Vinci’s originating work. The diagram shows the fairly obvious correspondences that would be drawn between the two visual depictions, but does not yet show the more significant driving force ‘behind’ the assumption that a blend is operating – for this, we need to spell out in more detail the properties and organizations assumed for the input spaces, the particular mappings that are selected, and the corresponding blended space. The content-related workings of the intermedial reference may be characterized more finely, for example, by taking the group of characters, their spatial relationships and respective properties as one input space, and the story and representation of the Last Supper as a second input space. The shared or generic space serving as a foundation for relating these spaces then needs to pick out entities, relations, and activities that are to be considered similar across the input spaces for the purposes of the blend. In the present case, this can be taken as a commonsense view of people, everyday objects, places, and spatial relationships together with some relevant roles and interpersonal relations, such as the social configuration of a leader with followers. It is the recognition (or hypothesizing) of this common structure that consequently supports blending. Combining the input spaces then gives (as one possibility) several blended individuals, relationships, and activities. The Last Supper as such as a kind of farewell is maintained and supported by both input spaces. Particular blends of characters are provided by virtue of the spatial relationships that are preserved. These bring with them particular modified roles from the Last Supper input space. This level of detail is necessary in order to identify more precisely just what the blend is providing because it would of course be possible to provide superficially similar correlations for many other films, TV series, paintings, and posters that have made use of this motif, ranging from the grouping of beggars in Luis Buñuel’s Viridiana (1961) to Homer and other assorted characters sitting in Moe’s Tavern in the final scene of the “Thank God It’s Doomsday” episode of Matt Groening’s The Simpsons (2005: S16 E19). Critical discussion of these occurrences often rate them very differently, however – some are considered pointless imitation, some as humorous associations, while others appear more motivated. The basis for such differences in judgments follows not from the bare fact of constructing an association as depicted in Figure 7.2, but rather from ‘running the blend’. It is only this latter step that allows the appropriateness and value of the blend to be assessed by examining the inferences that can be drawn from within the newly constructed local world that the blend constitutes. The more productive and appropriate those inferences appear to be for the broader communicative point of the originating artifact, the less likely the association will be seen as gratuitous. Discussions of the ‘meanings’ or ‘appropriateness’ of particular uses are characterizable in terms of the blend that has been applied.

Intermediality in Film

149

Figure 7.3 Running the blend to derive further interpretations of details from the input spaces, such as a ‘lamp ~ halo’ association and blended transfer of spiritual attributes.

Running the blend thus elaborates on the information in the input spaces so that further inferences may be applied and new structures can emerge. For example, without the additional inferences licensed by the Last Supper, the original shot from Mash would contain details lacking ‘deeper’ motivation. Blending with the Last Supper provides further directions of interpretation. These also stretch beyond more straight-forward associations between a leader and that leader’s assorted followers to include potential readings of less central features, such as the bright lamp that happens to be placed near the Mash character Captain ‘Painless’ Waldowski (played by John Schuck). As suggested in Figure 7.3, we can see that this figure ‘inherits’ the role and position of Jesus by virtue of his spatial placement. It is then possible to pursue a reading in which the bright lamp in Mash functions as a spiritual halo (even though this information is not present in the Last Supper). This shows how a tighter analytic focus on the nature of blending proves crucial. This example has suggested informally the central formal properties of the notion of blending; these more formal underpinnings are to be assumed as operating ‘behind the scenes’ in all the examples given below. The recognition of richly structured spaces that serve as inputs for structured mappings should have clarified in what ways a suitably refined formal notion of blending can take us beyond the simple fact of recognizing an intermedial reference. Successful blending relies entirely on the richness of the structures provided in the spaces being blended. Without structural correlates across those spaces, as picked out in the generic space, there is little for the mappings between spaces to grasp. It is in this sense that the mechanisms of blending offer far more than simple associations or repetitions of works, or portions of works, that may have been seen elsewhere.

150

John A. Bateman

3 Diverse Kinds of Intermedial Blending in Film The blend examples in the previous section were relatively straightforward. In this section we move more specifically to notions of intermedial ‘mixing’ in film, suggesting with examples how a treatment in terms of blending might be beneficial. For this, we will need to pursue expansions along two interlinked dimensions: that of temporality and that of discourse. With the discussion so far, and with most considerations of blending or metaphor-based interpretation offered in the literature, blending accounts can appear rather static. However, as Kappelhoff and Greifenstein emphasize in their adoption of ‘audio-visual metaphors’, for adequate treatments of film it is important to shift the focus more “to the dynamic elaboration of metaphor in discourse” (Kappelhoff/Greifenstein 2016: 190). This is precisely the direction that will now be taken for intermediality. The use of blending as a filmic device raises a number of specific ‘tasks’ that need to be managed dynamically, both by film-makers and film interpreters. The tasks include: • • • •

giving sufficient clues that there is something to blend: attention guidance, identifying the relevant (additional) input spaces to be incorporated: retrieval cues (cf., e.g., Veale 1997), identifying which portions of the input spaces are to be covered by the generic space: discourse cues, and identifying over what extent of the film the blended aspects are intended to endure: relevance management.

Only when these tasks are all successfully performed can a viewer be expected to compose input spaces to yield a blended space and thereby be able to run the blend over some relevant segment(s) of a film. Here we will focus particularly on the fundamental question of identifying when blends are intended, and once identified, address their extent, or ‘scope’: how does a film indicate that a blend is finished with? For indicating that a blend or blends may be relevant, the cases of the previous section already made use of distinct signaling methods. The blend involving the Last Supper in Mash relied on a highly distinctive distribution of characters in space, each adopting particular postures and gestures. In addition to this, however, it also drew on a more ‘pictorial’ or ‘painterly’ set of lighting decisions that were not motivated internally by the narrative at that point in the film and so invited further explanation. For those (many) viewers familiar with da Vinci’s Last Supper imagery, the visual association would be difficult to overlook. Performing the operations of blending is then made available as a resource for interpreting the film at that point. The film then does rather little with the ‘world’ created by the blend and so the effect is primarily one of humorously flavoring the atmosphere of the scene in which it is used. There are many further examples of distinct kinds of blending that

Intermediality in Film

151

combine this and other methods of inviting an application of blending. And, these in turn can be used to show that rather different aspects of films need to be considered as functioning in the blend. Identifying these will itself offer a useful diagnostic tool for teasing apart some of the representational layers necessary for more adequate film descriptions and analyses and for raising new research questions. To begin our more detailed discussion, therefore, let us consider the sequence of selected frames shown in Figure 7.4 taken from Andrew Adamson and Vicky Jenson’s Shrek (2001).2 The scene’s beginning involves Princess Fiona, the princess of the generic fable being parodied, taking offense at a singing Robin Hood figure and his troupe. Rather than acting in a more traditional princessly manner, she quickly renders ‘Monsieur Hood’, as he is called, unconscious before going on to incapacitate single-handedly the remaining members of the troupe – first,

Figure 7.4 A frame sequence from the ‘Robin Hood’ scene from Andrew Adamson and Vicky Jenson’s Shrek (2001, at approximately 0:50:00).

152

John A. Bateman

in frames 1–9, moving rapidly in a series of head over heels somersaults towards one of the troupe and, subsequently, dealing with the rest in equally acrobatic fashion. This is then taken to an extreme in frames 10–18, where the more direct following of the action by the ‘camera’ is replaced by a section where  Fiona, seeing two of Monsieur Hood’s troupe approaching from opposite directions, jumps into the air (frame 11) and then kicks them both simultaneously at head level (at frame 17). During this jumping and kicking, the camera revolves in a complete circle around the entire group, slowing the action at the same time. This is then a direct ‘quotation’ of the famous bullet-time shot from the beginning of the Wachowskis’ The Matrix (1999), where the main female character, Trinity, similarly disposes of an opponent. The surprise physical effect of this initial depiction arose from the fact that for a live action shot the camera would need to move around the actors at an impossibly high speed if it were to capture what is going on while the jumping Trinity is still in the air, thus raising the stakes beyond simple slow motion. The technique of stitching images taken from different physical cameras together that made the original bullet time sequences of The Matrix possible is now well known. More relevant for our current purposes is how this adds another type of explicit indicator for viewers that it may be relevant to seek and apply a blend. That is, it is not just the simple fact of jumping into the air in a martial arts fashion that serves here  to make the necessary connection, but the actual (although simulated) style of extreme camera movement that is involved. In fact, as we move into this sequence there were already a number of indicators that a move away from more ‘traditional’ animated style was imminent. The ‘camera’ is placed on fast moving objects (e.g., an arrow), for example, and specific and distinctive camera effects are simulated, such as differentially zoomed foreground and background. The scene also exhibits several further filmic references to actions found in The Matrix and other martial art films as well as additional quotations of other montage techniques – for example, a single kicking action marking the end of the fight is constructed out of three distinct depictions of the kick in a manner reminiscent of the famous plate smashing scene in Sergei Eisenstein’s 1925 film Battleship Potemkin (cf. Bordwell 1985: 85–87). All of these expand the generic norms potentially relevant for the film. For any viewers who have seen the distinctive scene from The Matrix, however, consideration of this specific invited blend will be unavoidable: Princess Fiona and Trinity will be blended, which then also naturally contributes to the construction of the Princess Fiona character. The question of how long and in what form this blend, once established, will remain relevant for the film is more interesting. On the one hand, there are few indications that the specific Fiona-Trinity (or Shrek-The Matrix) blend will remain sufficiently relevant to warrant ‘running the blend’ further. It is highly unlikely, for example, that a viewer will pursue, for example, readings in which the ogre Shrek takes on properties of Neo, the main male

Intermediality in Film

153

protagonist of The Matrix. This is because the film itself provides no further structural indications that the blend should be maintained. This then parallels Kappelhoff/Greifenstein’s (2016: 190) consideration of the emergence of metaphors as being essentially linked and dynamically constructed, as mentioned above, following the “multimodal structure of the unfolding film”. On the other hand, the techniques employed all add to the viewer’s understanding of Fiona’s character as possessing capabilities beyond that which might be expected of fairy tale princesses. This then ‘stretches’ the influence of the blend well beyond the immediate events – indeed, Unger and Sunderland, in their analysis of the Robin Hood episode, suggest that “this scene is more subversive of traditional gender stereotypes than any other in the whole film” (Unger/Sunderland 2007: 478). The blend, therefore, does more work than the momentary occurrence of the Last Supper in Mash since its contribution to the construction of Fiona’s character no doubt constitutes part of the position the film is taking to gender roles in general. Blends can then well extend an influence over much larger portions of a film than the moments in which they appear, and there are several approaches that attempt to model this. In Veale’s (1997) narrative dynamic blending model, for example, input spaces are considered from the perspective both of ‘genres’ and of ‘themes’ whose relevance persists in the form of a ‘workspace’ for possible blends while a narrative is being developed. In contrast, Kappelhoff and Greifenstein’s approach draws for a very similar effect on work on discourse realized multimodally in speech and gesture that has shown that cognitive metaphors emerge and operate over longer stretches of interaction (cf. Müller/Cienki 2009). We draw similarly on multimodal discourse research, but particularly on our own account of visual and audio-visual discourse organization since this has already been developed extensively for film (cf. Bateman/Wildfeuer 2014). Then, on the basis of an analysis of the film’s discourse organization, its division into scenes and other structural units, we obtain a structural ‘backbone’ that serves equally for temporally anchoring and developing any blends that may be activated. Our consideration of the Shrek example also strengthens our claim that there are dimensions, or kinds of structures, that need to be considered when characterizing blends activated in a film that lie beyond ‘content’-related issues. The use of certain technical features, such as camera movements and montage, for indicating relevant blends opens up further ‘intermedial’ references that may be given descriptions in terms of blends employing levels of abstraction quite different to those of content. When the bullet time technique was first seen widely in The Matrix, for example, it came to be associated with, and indicative of, a sophisticated and readily identifiable aesthetic. The film was consequently characterized as innovative, fashionable, and trend-setting in many respects. The fact of the achievement of such effects with live action was an important component of this (subsequently compromised by a more extensive use of CGI in the sequels). Shrek thus not only quotes the fact of Princess Fiona being able to defeat groups of opponents acrobatically, but also,

154

John A. Bateman

and more significantly, quotes the high-tech, fashionable aesthetic established for The Matrix. And, as a final step, we need to see this less as an ‘intellectual’ appraisal on the part of the viewer that references to other media may be involved and more as a ‘(psycho)physical’ placement of the viewer with respect to those other media. This will be taken up further when we discuss the relationships between blending and ‘embodiment’ below. Technically, of course, there is no problem constructing animated action that exhibits these kinds of hyper-slow motion and fast camera movement: animation has always made creative use of its independence of the constraints of live action. It is then no surprise that Shrek makes good use of this even while it is quoting the technical effects of The Matrix by pushing the situation beyond what would be physically realizable at all. During Princess Fiona’s jump, and in the middle of the simulated camera movement, Fiona pauses in mid-air to push back and arrange her hair. As she pauses, the camera movement is also suspended, only resuming its rotation when Fiona is ready to proceed. Unger/Sunderland (2007: 479) hesitantly suggest that this at least problematizes the proclaimed construction of Princess Fiona’s character against traditional gender stereotypes, although her character is a rather more complex combination of sources. It is, however, typical of Shrek that it includes many such ‘modified’ or exaggerated intermedial references, each contributing to an overall effect that positions the film (and the viewer’s appraisal of the film) as fresh and innovative. Moving beyond ‘content’-based blends requires that we now examine rather more closely just what is being blended and to what effect within filmic intermedial references. Consider, for example, the opening sequence of Paul Verhoeven’s Starship Troopers (1997), which takes the form of a news program reporting on the state of affairs in an ongoing war between the Earth and a collection of giant insect-like species in distant space. The news program conforms to a readily recognizable format of news media, complete with intercut logos, recruitment ‘ads’, interviews with a happy military, a cute child, interwoven generic scenes of flags and soldiers, mini-documentary pieces with dynamic infographics, and a reporter sending ‘live’ from the middle of a battlezone – which then gets somewhat out of hand as the reporter is cut in half ‘live’ on camera by the pincers of an attacking ‘bug’. Figure 7.5 shows representative frames.

Figure 7.5 Representative frames from the opening sequence of Paul Verhoeven’s Starship Troopers (1997).

Intermediality in Film

155

This sequence consequently exhibits several distinct kinds of blending in operation, each of which draws on distinct structural domains. First, there is the news media report itself and its generic structure (complete with subgenres and textual overlays, for example, informing the constructed viewer that they are watching ‘live’). Second, there is a suggested nod towards interactive media, since at transition points in the sequence upper and lower menu bars and a blinking arrow cursor appear on screen giving further topics and the question ‘Would you like to know more?’. Third, there are live reports as such, indicated by long-established technical features such as shaky handheld views. And fourth, also within the ‘live’ scenes, there are further indications of unedited live media given by intermittent disturbances in the image quality common for technical problems of media distribution in difficult situations. Since this entire sequence positions itself with documentary reporting, these domains can, in addition, be related to Nichols’ (2001: 99, 138) six documentary ‘modes of representation’. The sequence begins with ‘expository’ representations addressing the viewer directly with voice-overs, infographics, and the voice and image of the reporter during the ‘live’ segment at the end, separated by instances of ‘participatory’/‘performative’ (soldiers and children ‘doing their part’) and ‘poetic’ (views of soldiers and flags) modes in the middle. These constructed and controlled forms then give way to the inadvertently ‘observational’ fragment at the conclusion, when the reporter is no longer available but the camera is still recording. Here finally there is direct address, not to the audience but to the diegetic camera operator (as one soldier looks directly into the camera shouting “get out of here, now!”). All of these media elements are themselves depicted ‘as if’ they are being presented directly to the viewer on some multimedia news platform, which is quite different to the use of TV or computer screens diegetically within shots. In such cases, the viewer is made aware (at some point) that the film is showing a screen that is present in the story world; no such mediation occurs here. The principal media and genre placement of the storyline as an action film does, however, remain: as the ‘live’ news report moves towards its climax, orchestral film music of a fairly conventional kind becomes increasingly prominent – again, it is precisely the ongoing blend between media forms that allows this to be interpreted appropriately; using such music in an actual news report would (depending somewhat on culture) be more than odd. It has become relatively common, of course, to employ these kinds of frame-breaking depictions, importing media styles that were not traditionally part of film. Verhoeven’s use of the news program and ‘pseudo-documentary’ is an aesthetic and stylistic choice and there is no expectation that the audience will in any way be confused as to the fictional nature of what is being depicted. This makes it an interesting question to consider just what the function of such flourishes is. On the one hand, the exaggerated nature of the depictions might be ‘transferred back’ across the blend so as to offer a parodic

156

John A. Bateman

Figure 7.6 Frames from the congressional hearing scene from Jon Fravreau’s Iron Man 2 (2010).

appraisal of the news media input space. On the other, immediately ‘physical’ technical features, such as shaky handheld images are well established in quasi-first-person narratives or particular film segments – as in The Blair Witch Project (Myrick and Sánchez 1999), Cloverfield (Reeves 2008), Battle Of Los Angeles (Atkins 2011) and many more – and so might be seen as techniques for raising or suggesting unmediated involvement, although this can also be taken too far as argued, for example, by Provencher (2008). However, further cases appear not to be explained in these ways. Figure 7.6, for example, shows frames selected from the congressional hearing scene from Jon Favreau’s Iron Man 2 (2010). Here the text and ‘station identification’ overlaid at the bottom of the image ‘quotes’ similar hearings as seen on TV, but with no expectation that realism or confusion over the status of the depicted events should ensue. What does follow, however, is the incorporation of a range of interpretations for other decisions of film style and mise-en-scène that are employed. The appearance of text captions in frame is very restricted within narrative film – essentially limited to location and time information. Blending with the news channel medium allows text-in-frame to take on additional roles. Moreover, in the selected scene, people filming the scene are visible at certain points, and there is also a swish pan as one new character is (supposedly unexpected) called to testify. Again we can see how the blended-in news channel medium provides viewers with the resources and expectations necessary for interpreting these effects, interpretations which would be quite different if they occurred elsewhere in the film outside of the scope of a news channel blend. This, therefore, constitutes a small-scale media-blend, which may in addition have consequences for genre expectations. Taking this development further, in a manner similar to the opening sequence from Starship Troopers, but portrayed far more realistically and without parody, Doug Liman’s Edge Of Tomorrow (2014) begins with a sequence of news reports re-configured to tell the story of an alien invasion of Earth. Some of these news reports involve genuine footage removed from its original context; others use genuine news reporters who would be known to at least some of the viewers. Figure 7.7 shows representative frames similar to those shown above for Starship Troopers. In this case, due to the realistic nature of the images selected, there may well be support for a limited ‘suspension of disbelief’ helped by the very direct quotations of news

Intermediality in Film

157

Figure 7.7 Representative frames from the opening sequence of Doug Liman’s Edge Of Tomorrow (2014).

media and those quotations’ evident “semblance of truth” (cf. Coleridge 1817: Chap. XIV) – at least perhaps until Tom Cruise appears as one of the interviewed army officers in the news reports (bottom row, last two frames). For media blends of this kind, this relation between viewer and material begins to take on a particularly central role. When we consider the structural domains that are being blended, we require not a content-related structural organization but instead a characterization of the communicative situation in which the medium is typically present. This includes structural assumptions concerning the relation between the depicting material and that which is depicted (e.g., live, indexical, edited vs. unedited, and so on), which is then particularly interesting in terms of the different ‘truth claims’ that may be constructed. In most of the cases shown so far, there could be little doubt that we are dealing with fictional narrative film and so the media construction might not be expected to involve any claims of veracity. But already in the case of Edge Of Tomorrow, the situation becomes less clear-cut. Captions for identifying fictional events of the kind we saw in Iron Man 2 are also employed here but now overlaid on material that is most likely actual news footage. One of these (Figure 7.7, top row, center frame), for example, shows a meteor trail together with the text caption “Panic as meteor strikes Germany” (additionally sourced fictionally to Al Jazeera); the images appear to be the actual amateur video recordings made of the Chelyabinsk meteor over the Urals in February 2013. The frame following shows a crowd in a panic with the text caption “Hamburg 1 Aktuell” (complete with time and temperature captions). The status of these images and their accompanying texts is then ambiguous: for the viewer who cannot identify the original sources, the question is whether the images have been taken together with their original captions in order to fit the storyline or whether the images are ‘real’ and the captions fictional. Only when the content is more explicitly anchored to the evidently fictional storyline of invading aliens does this potential ambiguity become resolvable with any confidence.

158

John A. Bateman

Figure 7.8 Representative frames from the production company logo sequence at the beginning of Liman’s Edge Of Tomorrow (2014).

‘Reality’ markers are also offered by the depiction of media-specific technical breakdowns familiar from live media such as TV and news reportage. This technique is now quite common in fictional films and often cooccurs with other technical features such as handheld views. The technique has also extended the places where it might occur, however, thus pushing this particular blending of media forms a step further. In the present case, the opening seconds of the film displaying the respective logos and trademarks of the involved production companies are also shown as exhibiting technical difficulties and problems comparable to those that might be experienced in live reporting over unstable communication channels. Specifically: after a few seconds, the opening “Warner Bros. Pictures (WB)” credit logo is distorted and disappears off screen, followed by some indeterminate distorted frames reminiscent of a non-digital TV losing the signal. The ensuing “Village Roadshow Pictures” logo is introduced similarly degraded. Finally, a news anchor person, shown evidently not sure whether he is on air or not, is intercut with the final distorted logo for “RatPac-Dune Entertainment”. The sound track is similarly fragmented and degraded throughout the entire sequence. Representative frames can be seen in Figure 7.8. The blend here is particularly interesting. On the one hand, the audience knows that it is watching a completely fixed, edited film, with very little chance of actual technical difficulties intervening between the medium, the projector, and the screen. On the other hand, the material as presented identifies itself as a medium fully susceptible to the vagaries of uncertain transmission channels. The structures that are blended here are therefore far removed from depictions of content or story worlds and target the actual situations of consumption and production of a film instead, drawing these into the depicted material. This transferal of production and distribution properties to an artifact’s resources for signifying is itself well-known and can also be observed across many media: such as when the surface noise and scratches of earlier generations of vinyl records are now produced deliberately as part of the texture of digitally produced music, (cf. Kress/van Leeuwen 2001: 95). In film, this effect can also be readily observed, as in examples such as Guy Maddin’s The Saddest Music In The World (2003), where grainy and often far

Intermediality in Film

159

Figure 7.9 Suggestive blend diagram involving the mediated communicative situations of film and live broadcast news.

from clear black and white or tinted images combine with low-resolution sound to considerable effect. An interesting difference in the case focused on here, however, is how the media structures being blended involve live media but extended to the supposedly non-diegetic, ‘paratextual’ (Genette 1997) parts of the film as well. The blend in cases such as The Saddest Music In The World, in contrast, is a more straightforward one of the filmic technical resources at different time periods, which does not place the audience in a fundamentally different relation to the material being viewed. The effect is also rather different to the now relatively common ‘theming’ of company logos and credits in opening sequences to reflect more the genre or atmosphere of the film that follows. This different relation is suggested in terms of a media blend in Figure 7.9, echoing the blend diagrams introduced above but now addressed to the mediated communication situation rather than content. The idea here is to record and document the source of particular uncertainties as well as potential interpretation strategies that might apply within the scope of the blend. The input spaces are activated by the technical features of the medium that the viewer notices, such as breaks in transmission or recognized news formats. The blend at the top of the diagram must then take on the task of reconciling these potentially conflicting indications, which may well leave open questions concerning the factual (or otherwise) nature of depicted events, the degree of editing, and so on. The blend is thus a ‘theory’ of the communicative situation in which the viewer is attempting to place him or herself. We noted above how blending and conceptual metaphor theory are now commonly related to issues of cognition, particularly embodied cognition.

160

John A. Bateman

This typically operates in terms of Lakoff/Johnson’s (1999) notion of ‘image schemas’. Image schemas are constellations of embodied motor and perceptual experiences acquired in the early years of life, which then serve as basic building blocks for constructing more abstract structures employing processes similar to blending. In a sense, such image schemas can be seen as blending diagrams where the generic space or one or more of the input spaces are not abstract, conceptual representations but direct motor/ perceptual, or ‘bodily-formatted’, representations (cf. Gallese/Guerra 2012: 193). This is what offers much of the neurobiological foundation necessary for Fauconnier/Turner’s (2003) mental space model. Now, if we consider a notion of an embodied communicative situation, characterizations of the communicative situation may well impinge on other aspects of processing at quite a deep level. It is known, for example, that the attribution of either fictional or real status to visual depictions can affect viewers’ emotional and cognitive processing (e.g., Mendelson/Papacharissi 2007). ‘Understandings’ of the media situations in which viewers see themselves may then well prove relevant for a range of responses. To the extent that attributions of reality draw on media cues, therefore, constructing artifacts whose media statuses are themselves problematic or ‘blended’ may well give rise to further effects that need to be addressed in recipient studies. Media blends of this kind require far closer attention, particularly in cases where the contents are not so clearly identifiable as fictional or, even more complex, when (aspects of) the contents are explicitly not fictional. To illustrate a case where the communicative situation is constructed to be not at all clear, we take the opening sequence of Ron Howard’s Frost/Nixon (2008) as a final example. The blending here is similar to that of Edge Of Tomorrow with the exception that the depicted events are principally intended to be (at least based on) real events. The film tells the story of the planning and execution of the series of TV interviews conducted between David Frost and former US president Richard Nixon in 1977, post-Watergate and Nixon’s resignation from office. Selected frames from the opening sequence are shown in Figure 7.10. The sequence, which lasts almost 7 minutes, begins with genuine archive news material exhibiting period TV visual and sound degradation intercut with fragmentary, often blurred but nevertheless high visual production quality shots of what we soon learn to be Nixon preparing for his final televised speech from the White House in which he announces his resignation. After the first words of his speech, the sequence cuts to a first person report from one of the main protagonists of the film (James Reston, Jr. played by Sam Rockwell: fourth row, frame 3), shot with an ‘in-between’ visual quality of slightly washed out, side-lit daylight colors and reduced tonal variation. The character relates how the speech left him feeling ‘angry’ and ‘disappointed’ because there was no sign of remorse or apology from Nixon, thus setting up the main narrative arc for the film. After a few such talking head shots with characters who will play roles in the film, the sequence turns (fifth row,

Intermediality in Film

161

Figure 7.10 Opening sequence from Ron Howard’s Frost/Nixon (2008).

frame 2) to introducing David Frost (played by Michael Sheen), hosting his regular TV show at that time on Australian TV – this time shot with more regular film production values. The show is depicted ending just as Richard Nixon gives his speech live on TV. There are several shots of Frost watching the speech, which is shown with the degraded period TV visual effects seen earlier for the genuine footage, but featuring Frank Langella, the actor playing Nixon. Several diegetic views of TV screens appear during the opening sequence; these are often shown in the  media style of period TV but are filmed with the actors of the film in the manner of ‘fake’ documentary material popularized particularly in earlier films such as Woody Allen’s Zelig (1983) or Robert Zemeckis’ Forrest Gump (1994). The gradual transition from documentary or archive footage to the main body of the film is handled particularly smoothly in this sequence. The montage constructs this transition at one level, cross-cutting actual and fictional depictions; even within the depictions, however, there are mixtures of media either occurring or suggested by differential image and sound quality. In addition, the way in which several of the main characters are introduced by means of realistically portrayed interviews supports an uncertain boundary between the narrative of the film and actual reported events. It is necessary to consider the effects of such constructed medium uncertainty or mixing. If it turns out that viewers are not able to completely insulate their responses to the film that follows from the very deliberate anchoring of that film in historical events, both in terms of content and

162

John A. Bateman

presentational style, then this would be an important ethical consideration for the design choices made. No film can, of course, present a value-free or ‘objective’ rendering of its subject matter and so the directions of interpretative construction performed need to be scrutinized; an issue taken up explicitly with respect to Frost/Nixon by Denham (2010) in Media Ethics. In fact, on the critical ‘blame’ and ‘legality’ issues, it can be argued that the view of Richard Nixon offered in the film constructs him in a rather more positive light than the recordings of the actual interview. An important question would then be the extent to which the media blend encouraged by the film’s organization destabilizes the ‘felt’ communicative situation so as to bring about more positive judgments of veracity than would have been the case without the framing media blends. Such an effect may well be strengthened by the association drawn between blends and embodiment noted above.

4 Conclusion and Outlook Notions of productive ‘mixing’ have long been a recurring motif in discussions of filmic meaning-making. In this chapter, we have suggested how a formalization of the cognitive science view of blending may offer a mechanism capable of generating, and hence describing, ‘mixes’ at various levels in the aesthetic construction of film. The mechanism itself is very general and may be applied to quite diverse phenomena relevant for film. Previously, for example, blending has been considered for treatments of genre and narrative mixing (Veale 1997) and for the construction and identification of characters or figures in film (Bacon 2009). There are also further areas that might be addressed: a quite different area of application, for example, would be to Livingston’s (2013) examples of diegetic characters interacting with on-screen credits or other textual inscriptions. The kinds of interaction between multiple sources and levels (diegetic/non-diegetic) found here all appear ripe for more systematic blending-based accounts. The examples focused on in this present chapter have considered specifically what may be described as media blends. There is increasing theoretical debate concerning the general phenomena of intermediality, transmediality, and media convergence (cf., e.g., Bolter/Grusin 2000; Paech 2003; Rajewsky 2005; Wolf 2007; Elleström 2010) and so further mechanisms to support such studies would be beneficial. Blending offers a rich way of formalizing relationships across distinct filmically-relevant domains but also requires us to be more explicit about the structural organization of any domains being combined. Consequently, we have explored here the formal characteristics of intermedial blending to suggest a method for uncovering some of the structural configurations that need to be acknowledged and developed further for explaining these specific intertextual and intermedial relationships in film. Some of these may then be conventionalized as genres, thus productively combining considerations of media and genre.

Intermediality in Film

163

Finally, although blending offers a foundation for describing how differently structured information from distinct domains may be productively merged, there is a further aspect necessary for any complete picture of blending that is equally important when considering film and media blends. As noted above, blending as such only indicates how information may be brought together: it does not of itself explain precisely which blends are created during production or interpretation – that is, precisely which input domains are made to contribute, and which generic spaces are employed, is not fixed by the mechanism itself. For this, mention is commonly made in the blending literature of ‘finding the story’ – that is, notions of narrative are taken to influence (somehow) the operations of the basic mechanisms of blending. This background leads Forceville to adopt a cautious tone concerning blending, emphasizing the importance of additional ‘pragmatic’ considerations “involving a goal” when selecting blends (cf. Forceville 2013: 257). And, similarly, Veale et al. insist: “Blending theory cannot be considered a true theory of producer-centric creativity until it can explicitly identify the heuristics, pathways, and mechanisms that allow a producer to infer the contents of a second input space for a given input in a specific goal-oriented context.” (Veale et al. 2013: 49) Although various sources can be explored for providing such constraints, in our discussion above we have oriented specifically to the fine-grained details of the unfolding film as providing more or less explicit indicators of just which blends are to be pursued. This in turn provides a natural bridge to the more general notions of text and textuality considered throughout this volume. Textuality, of which narrative is just one form, can as a consequence also be considered from the perspective of its contribution to the selection and maintenance of blends. Under this view, it is what a text is trying to do or achieve with some, potentially new, media blend that drives which blends which will be pursued and which will not – an orientation that fits well with several existing proposals made for the development both of film and of film as a medium.

Acknowledgements Thanks are due to Chiao-I Tseng and Janina Wildfeuer for comments on a previous version of this chapter, as well as to feedback from participants during presentations at the 2015 meeting of the Society for Cognitive Studies of the Moving Image (SCSMI) at Birkbeck College, London, and at the Image schema workshop held at the KRDB Research Centre for Knowledge and Data of the Free University of Bozen-Bolzano in October 2015.

164

John A. Bateman

Notes 1. In the cognitive blending literature these diagrams are typically shown the other way round, with the generic space at the top and the blend at the bottom. For reasons related to our formalization and the use of particular mathematical techniques for the modeling of blends, the diagrams we employ orient to a different tradition. 2. Here and in all subsequent composite figures, the frames are to be read left-toright, top-to-bottom.

Bibliography Allen, M. (2013). “Against ‘Hybridity’ in Genre Studies: Blending as an Alternative Approach to Generic Experimentation”. Trespassing Journal: An Online Journal of Trespassing Art, Science, and Philosophy 2. ISSN: 2147-2134. Altman, R. (1999). Film/Genre. London: British Film Institute. Bacon, H. (2009). “Blendings of Real, Fictional and Other Imaginary People”. Projections: Journal for Movies and Mind 3(1), 77–99. Bateman, J. A. (2016). “Methodological and Theoretical Issues for the Empirical Investigation of Multimodality”. In: Klug, N.-M./Stöckl, H. (eds.). Sprache im multimodalen Kontext/Language and Multimodality. Handbooks of Linguistics and Communication Science (HSK), Berlin: de Gruyter Mouton. Bateman, J. A./Wildfeuer, J. (2014). “A Multimodal Discourse Theory of Visual Narrative”, Journal of Pragmatics 74: 180–218. Online: http://dx.doi.org/10.1016/j. pragma.2014.10.001 [last accessed: 01 March 2016]. Black, M. (1962). “Metaphor”. In: Black, M. (ed.). Models and Metaphors. Ithaca: Cornell University Press, pp. 25–47. Bolter, J. D./Grusin, R. (2000). Remediation: Understanding New Media. Cambridge: MIT Press. Bordwell, D. (1985). Narration in the Fiction Film. Madison: University of Wisconsin Press. Bordwell, D. (2006). The Way Hollywood Tells It: Story and Style in Modern Movies. Berkeley/Los Angeles: University of California Press. Bordwell, D./Thompson, K. (1979). Film Art: An Introduction. New York: Knopf. Coëgnarts, M./Kravanja, P. (2012). “Embodied Visual Meaning: Image Schemas in Film”. Projections: Journal for Movies and Mind 6(2), 84–101. Coleridge, S. T. (1817). Biographia Literaria; or Biographical Sketches of my Literary Life and Opinions. London: Rest Fenner, 23, Paternoster Row. Coulson, S. (2001). Semantic Leaps: Frame-Shifting and Conceptual Blending in Meaning Construction. Cambridge, UK: Cambridge University Press. Denham, B. E. (2010). “Frost/Nixon: Historical Accuracy and Press/Government Relations”. Media Ethics 21(2). Online: http://www.mediaethicsmagazine.com/ [last accessed: 01 March 2016]. Derrida, J. (2001[1978]). Writing and Difference. London: University of Chicago Press. Elleström, L. (2010). “The Modalities of Media: A Model for Understanding Intermedial Relations”. In: Elleström, L. (ed.). Media Borders, Multimodality and Intermediality. Basingstoke: Palgrave Macmillan, pp. 11–50. Fauconnier, G. (1997). Mappings in Thought and Language. Cambridge: Cambridge University Press.

Intermediality in Film

165

Fauconnier, G./Turner, M. (2003). The Way We Think: Conceptual Blending and the Mind’s Hidden Complexities. New York: Basic Books. Forceville, C. J. (2009). “Non-Verbal and Multimodal Metaphor as a Cognitivist Framework: Agendas for Research”. In: Forceville, C.J./Urios-Aparisi, E. (eds.). Multimodal Metaphor. Berlin/New York: Mouton de Gruyter, pp. 19–42. Forceville, C. J. (2013). “Creative Visual Duality in Comics Balloons”. In: Veale, T./ Feyaerts, K./Forceville, C.J. (eds.). Creativity and the Agile Mind: A Multi-Disciplinary Study of a Multi-Facetted Phenomenon. Berlin: De Gruyter Mouton, pp. 253–273. Forceville, C. J. (2016). “Visual and Multimodal Metaphor in Film: Charting the Field”. In: Fahlenbrach, K. (ed.). Embodied Metaphors in Film, Television, and Video Games: Cognitive Approaches. New York/London: Routledge, pp. 17–32. Gallese, V./Guerra, M. (2012). “Embodying Movies: Embodied Simulation and Film Studies”. Cinema: Journal of Philosophy and the Moving Image 3: 183–210. Genette, G. (1997). Paratexts: Thresholds of Interpretation. Cambridge: Cambridge University Press. Originally published in French in 1987. Gentner, D. (1983). “Structure-Mapping: A Theoretical Framework for Analogy”. Cognitive Science 7: 155–170. Goguen, J. (2003). “Semiotic Morphisms, Representations and Blending for Interface Design”. In: Proceedings of the AMAST Workshop on Algebraic Methods in Language Processing. AMAST Press, pp. 1–15. Goguen, J. A./Harrell, D. F. (2010). “Style: A Computational and Conceptual Blending-Based Approach”. In: Argamon, S./Burns, K./Dubnov, S. (eds.). The Structure of Style: Algorithmic Approaches to Understanding Manner and Meaning. Berlin/Heidelberg: Springer, pp. 291–316. Gunkel, D. J. (2016). Of Remixology: Ethics and Aesthetics After Remix. Cambridge: Massachusetts Institute of Technology. Gust, H./Kühnberger, K.-U./Schmid, U. (2003). “Metaphors and Anti-Unification”. In: Proc. Twenty-First Workshop on Language Technology: Algebraic Methods in Language Processing, Verona, pp. 111–123. Jaffe, I. (2008). Hollywood Hybrids: Mixing Genres in Contemporary Films. Lanham: Rowman and Littlefield. Kappelhoff, H./Greifenstein, S. (2016). “Audiovisual Metaphors: Embodied Meaning and Processes of Fictionalization”. In: Fahlenbrach, K. (ed.). Embodied Metaphors in Film, Television, and Video Games: Cognitive Approaches. New York/ London: Routledge, pp. 183–201. Kappelhoff, H./Müller, C. (2011). “Embodied Meaning Construction. Multimodal Metaphor and Expressive Movement in Speech, Gesture, and in Feature Film”. Metaphor and the Social World 1(2), 121–153. Kress, G./van Leeuwen, T. (2001). Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold. Kutz, O. /Bateman, J. A./Neuhaus, F./Mossakowski, T. /Bhatt, M. (2015). “E pluribus unum: Formalisation, Use-Cases, and Computational Support for Conceptual Blending”. In: Besold, T.R./Schorlemmer, M./Smaill, A. (eds.). Computational Creativity Research: Towards Creative Machines. “Atlantis Thinking Machines”. Springer, pp. 167–196. Online: http://link.springer.com/ chapter/10.2991/978-94-6239-085-0_9 [last accessed: 01 March 2016]. Lakoff, G./Johnson, M. (1980). Metaphors We Live By. Chicago: Chicago University Press.

166

John A. Bateman

Lakoff, G./Johnson, M. (1999). Philosophy in the Flesh: The Embodied Mind and its Challenge to Western Thought. New York: Basic Books. Lefait, S./Ortoli, P. (eds.) (2012). In Praise of Cinematic Bastardy. Newcastle upon Tyne: Cambridge Scholars Publishing. Liebrand, C./Steiner, I. (eds.) (2004). Hollywood Hybrid: Genre und Gender im zeitgenössischen Mainstream-Film. Marburg: Schüren. Livingston, P. (2013). “The Imagined Seeing Hypothesis Film”. Projections: Journal for Movies and Mind 7(1), 139–146. Mendelson, A. L./Papacharissi, Z. (2007). “Reality vs. Fiction: How Defined Realness Affects Cognitive and Emotional Response to Photographs”. Visual Communication Quarterly 14(4), 231–243. Müller, C./Cienki, A. (2009). “Words, Gestures, and Beyond: Forms of Multimodal Metaphor in the Use of Spoken Language”. In: Forceville, C.J./Urios-Aparisi, E. (eds.). Multimodal Metaphor. Berlin/New York: Mouton de Gruyter, pp. 297–328. Neale, S. (1990). “Questions of Genre”. Screen 31(1), 45–66. Neale, S. (2000). Genre and Hollywood. London: Routledge. Nichols, B. (2001). Introduction to Documentary. Bloomington: Indiana University Press. Paech, J. (2003). “Intermedialität des Films”. In: Felix, J. (ed.) Moderne Film-Theorie. Mainz: Bender, pp. 287–312. Pereira, F. C. (2007). Creativity and Artificial Intelligence: A Conceptual Blending Approach. Applications of Cognitive Linguistics (ACL). Berlin: Mouton de Gruyter. Provencher, K. (2008). “Redacted’s Double Vision”. Film Quarterly 62(1), 32–38. Rajewsky, I. O. (2005). “Intermediality, Intertextuality, and Remediation: A Literary Perspective on Intermediality”. Intermédialités 6, 43–64. Schweinitz, J. (2002). “Von Filmgenres, Hybridformen und goldenen Nägeln”. In: Sellmer, J./Wulff, H.-J. (eds.). Film und Psychologie — Nach der kognitiven Phase? Schriftenreihe der Gesellschaft für Medienwissenschaft. Marburg: Schüren, pp. 79–92. Sonvilla-Weiss, S. (ed.) (2010). Mashup Cultures. Wien/New York: Springer. Staiger, J. (2007). “Hybrid or Inbred? The Purity Hypothesis and Hollywood Genre History”. In: Grant, B.K. (ed.). Film Genre Reader III. Austin: University of Texas Press, pp. 185–199. Thompson, K. (1999). Storytelling in the New Hollywood: Understanding Classical Narrative Technique. Cambridge: Harvard University Press. Turner, M. (2014). The Origin of Ideas: Blending, Creativity, and the Human Spark. Oxford: Oxford University Press. Unger, J. W./Sunderland, J. (2007). “Gendered Discourses in a Contemporary Animated Film: Subversion and Confirmation of Gender Stereotypes in Shrek”. In: Fairclough, N. /Cortese, G./Ardizzone, P. (eds.). Discourse and Contemporary Social Change. Bern/New York: Peter Lang, pp. 459–486. Veale, T. (1997). “Creativity as Pastiche: A Computational Model of Dynamic Blending and Textual Collage, with Special Reference to the Use of Blending in Cinematic Narratives”. Technical Report. Glasnevin, Dublin: School of Computer Applications, Dublin City University. In: Cinematic Narratives, presented at ICLC’97, the 1997 conference of The International Cognitive Linguistics Association. Online: http://afflatus.ucd.ie/Papers/mind1997.pdf [last accessed: 01 March 2016].

Intermediality in Film  167 Veale, T. Feyaerts, K./Forceville, C. J. (2013). “E Unis Pluribum: Using Mental Agility to Achieve Creative Duality in Word, Image and Sound”. In:Veale, T./Feyaerts, K./ Forceville, C.J. (eds.). Creativity and the Agile Mind: A Multi-Disciplinary Study of a Multi-Facetted Phenomenon. Berlin: De Gruyter Mouton, pp. 37–58. Wolf, W. (2007). “Description as a Transmedial Mode of Representation: General Features and Possibilities of Realization in Painting, Fiction and Music”. In: Wolf, W./Bernhart, W. (eds.). Description in Literature and Other Media. “Studies in Intermediality”. Amsterdam: Rodopi, pp. 1–90.

Filmography Alien (1979). Ridley Scott. Brandywine Productions/Twentieth Century-Fox ­Productions. USA/UK. Aliens (1986). James Cameron. Twentieth Century Fox Film Corporation/Brandywine Productions/SLM Production Group. UK/USA. Battle Of Los Angeles (2011). Jonathan Liebesman. Columbia Pictures/Relativity Media/Original Film. USA. Battleship Potemkin (1925). Sergei M. Eisenstein. Goskino/Mosfilm. Soviet Union. Blade Runner (1982). Ridley Scott. Ladd Company/Shaw Brothers/Warner Bros./ Blade Runner Partnership. USA/Hong Kong/UK. Blair Witch Project, The (1999). Daniel Myrick and Eduardo Sánchez. Haxan Films. USA. Close Encounters Of The Third Kind (1977). Steven Spielberg. Columbia ­Pictures Corporation/EMI Films/Julia Phillips and Michael Phillips Productions. USA. Cloverfield (2008). Matt Reeves. Paramount Pictures/Bad Robot. USA. Cowboys & Aliens (2011). Jon Favreau. Universal Pictures/DreamWorks SKG/ Reliance Entertainment/Relativity Media and others. USA. Edge Of Tomorrow (2014). Doug Liman. Warner Bros./Village Roadshow ­Pictures/ RatPac-Dune Entertainment and others. USA/Canada. Forrest Gump (1994). Robert Zemeckis. Paramount Pictures. USA. Frost/Nixon (2008). Ron Howard. Universal Pictures/Imagine Entertainment/ Working Title Films/StudioCanal and others. USA/UK/France. Iron Man 2 (2010). Jon Favreau. Paramount Pictures/Marvel Entertainment/­Marvel Studios/Fairview Entertainment. USA. Mash (1970). Robert Altman. Aspen Productions/Ingo Preminger Productions/ Twentieth Century Fox Film Corporation. USA. Matrix, The (1999). Lana Wachowski/Lilly Wachowski. Warner Bros./Village Roadshow Pictures/Groucho II Film Partnership/Silver Pictures. USA/Australia. Player, The (1992). Robert Altman. Avenue Pictures Productions/Spelling Entertainment/Addis Wechsler Pictures/Guild. USA. Pride and Prejudice and Zombies (2016). Burr Steers. Cross Creek Pictures/ MadRiver Pictures/QC Entertainment/Allison Shearmur Productions/Handsomecharlie Films/Head Gear Films. USA/UK. Saddest Music In The World, The (2003). Guy Maddin. Rhombus Media/Buffalo Gal Pictures/Ego Film Arts and others. Canada. Shrek (2001). Andrew Adamson/Vicky Jenson. DreamWorks Animation/DreamWorks SKG/Pacific Data Images (PDI). USA.

168  John A. Bateman Simpsons, The (1989-). James L. Brooks/Sam Simon/Matt Groening. Gracie Films/20th Century Fox Television/The Curiosity Company. USA. Starship Troopers (1997). Paul Verhoeven. TriStar Pictures/Touchstone Pictures/ Big Bug Pictures/Digital Image Associates. USA. Viridiana (1961). Luis Buñuel. Unión Industrial Cinematográfica (UNINCI)/Gustavo Alatriste/Films 59. Spain/Mexico. Zelig (1983). Woody Allen. Orion Pictures. USA.

8

Eat, Pray, LovE Expanding Adaptations and Global Tourism Joyce Goggin

1 Introduction: Adaptation Studies from Text to Film and Beyond Imitation and adaptation have been around for a very long time indeed – arguably at least since Aristotle’s discussion of mimesis as the backbone of poïesis in ‘The Poetics’. Having been taken more or less for granted as a kind of base condition for many forms of representational art since Aristotle, imitation as adaptation developed for centuries without much comment. Indeed, as Linda Hutcheon explains, “[t]he Victorians were great adapters of just about everything—and in just about every possible direction; the stories of poems, novels, plays, operas, paintings, songs, and tableaux vivants were constantly being adapted from one medium to another and then back again,” without anyone feeling the need to develop a specialized theory of adaptation (Hutcheon 2006: xi). Similarly, Cohen opens his work on the adaptation of fiction into film by remarking that “[t]he twentieth century began in a flurry of artistic hybrids, everything from calligrams to tone poems,” and while that century “put more rigorously into practice than ever before certain theories concerning the interrelatedness of the arts”, there persisted, according to Cohen in his ‘Film and Fiction’, a “dearth of discussion”, on how novels have been adapted for the cinema (Cohen 1979: 1, 2). While Cohen was certainly correct in lamenting the paucity of work on the topic, the systematic study of adaptation (the processes it necessitates and the industries involved, as well as the aesthetics, rhetoric, and historical periodization of which adaptations are composed) was, however, initiated by George Bluestone in his ‘Novels into Film: The Metamorphosis of Fiction into Film’ (1957). Following Bluestone’s watershed text and Cohen’s more theoretical study of fiction and film two decades later, adaptation began taking on increasing prominence in the academe in the 1980s and 1990s. Importantly, the study of adaptation rose to visibility, as both Bluestone’s and Cohen’s titles suggest, most notably in film and literature departments. That this interest in the topic would come from film and literature departments is certainly no surprise, given that adaptation studies belong implicitly to the institutional implementation of interdisciplinarity whereby, in the present case, it is assumed that a particular narrative property (text) is adapted into another medium (film), both of which media ‘belong’ to

170

Joyce Goggin

specific disciplines, making the properties in question germane to the coapplication of disciplinary methods. In a broader sense, adaptation studies were also informed by the vogue in, as well as the politics of, the postmodern deconstructionist drive to destabilize structuring practices such as the division of universities into single-discipline departments. So while this same decentering impulse led to merging departments like literature and film, and, in many cases their methods, on a broader scale the more philosophical operation of deconstructing binaries extends to opposing categories such as copy/original, author/ reader, and producer/viewer. In step then with the institutional expediency of creating interdisciplinary programs in literature and film on the one hand, and the politics, economics, and aesthetics of postmodernism that come into view in the 1970s on the other, the institutionalization of adaptation began to emerge in the 1980s and with it an every-growing body of scholarship.1 This is not to say, however, that the study of adaptation as a (quasi-) discipline has enjoyed a consensus as to what it is, or should be, and has progressed without debate on its objects and methods. Indeed, articles and books constantly appear claiming that adaptation studies are at a crossroads (cf. Leitch 2008), or are being contested, revised, revisited, or forced to take new directions (cf. Bruhn et al. 2013). Indeed, right from the get-go, adaptation studies have encountered a number of problems and issues, the most fundamental of which is perhaps the received notion that the ‘book was better than the movie’. This supposedly standard assessment of film adaptations is shorthand for the public’s perceived predilection for ‘originals’ that inheres stubbornly in the face of postmodernism and its characteristic deconstruction of various categories and binaries including the notion of fixed antecedents or original works that generate lesser copies or adaptations. In other words, the idea that the filmed adaptations of narratives, literary or otherwise, that first appeared as text, are somehow impoverished, place the text (chronologically and in terms of cultural capital) problematically ahead of the film, as implied by titles such as ‘Novels into Film’ (Bluestone 1957). As many scholars – perhaps most notably Kamilla Elliott (2003) – have argued, any judgment that hinges on the contention that one outing of a story is the original, while the adaptation is a sort of devalued copy, brings with it the issue of fidelity. While critics have decried the use of words like ‘fidelity’ and ‘faithfulness’ to the ‘original’ text for any number of reasons, the most obvious of which being how these terms automatically discredit the adaptation as being somehow untrue or promiscuous, Elliott compiled a list of perceptions of adaption that naturally follow from worrying about fidelity. Her list of six concepts takes in a range of fallacious notions concerning the purpose of adaptation beginning with the “psychic concept” whereby the adaptation is validated as being able to reproduce the “spirit” of the text, to the “trumping concept” through which a particular adaptation is seen as

Eat, Pray, Love

171

having gone the text one better, frequently in the area of period accuracy (Elliott 2003: 173–180). Similarly in an effort to head off value judgments at the pass, scholars like Dudley Andrew and Robert Stam devised categories based on how films adapt texts, or which features of a text they might be adapting. In his early work on these questions Dudley Andrew noted that the “analysis of adaptation must […] point to the achievement of equivalent narrative units in the absolutely different semiotic systems of film and language”, so that the products of adaptation should not be viewed as inferior, but rather, as different products that must be assessed on their own criteria (Andrew 1984: 14). Andrew, therefore, refers to types of adaptation such as “intersection” where “the uniqueness of the original text is preserved […] and left unassimilated in adaptation”, and he cites Bazin as describing certain modes of adaptation as “refractions” whereby the “original” text is a crystal chandelier that casts light in multitudinous directions, whereas the filmed adaptation is like a “flashlight interesting not for its own shape or the quality of its light but for what it makes appear in this or that dark corner” (Andrew 1984: 11). Robert Stam argued that “‘fidelity’ is an inadequate trope” and proposes other terms that might better account for the “inevitable losses and gains” involved in any kind of adaptation, suggesting terms such as “translation, reading, dialogization, cannibalization, transmutation, transfiguration, and signifying – each of which sheds light on a different dimension of adaption,” with the proviso that any text has the potential to “generate any number of critical readings and creative misreadings” (Stam 2000: 62–63). Deborah Cartmell and Imelda Whelehan likewise put forward the categories of “transposition” (such as the transposition of Shakespeare’s ‘Romeo and Juliet’ into a contemporary setting in Baz Luhrman’s 1996 film adaptation); “commentary” (such as Mira Nair’s Vanity Fair (2004), which commented on the politics of the source text by foregrounding the arguably underrepresented colonial origins of the wealth that drives Thackeray’s novel of 1847–48); and “analogue” (where enjoying the film does not necessitate previous knowledge of the source text, hence viewers of Amy Heckerling’s Clueless (1995) need not have read Austen’s ‘Emma’). In sum then, adaptation scholars have found it useful and productive to shift the focus away from shortfall in areas such as character development or plot detail in films that last only a few hours, when comparing them with the same narrative elements recounted in the expansive form of a novel of several hundred pages. As beneficial as various attempts to label, categorize or otherwise to parse out adaptation and the many practices it involves may be, this gesture too has several ramifications which I would like to point out before leaving this section on problems in adaptation studies. The theoretical groundwork laid down by scholars working in the area has provided us with an archive of approaches that is seemingly equipped to deal with adaptations of all kinds – from adaptations that are slavishly ‘faithful’ to the tutor text or which change little in the text in order to bring it to the screen, to those

172

Joyce Goggin

that preserve only some small detail such as the title.2 However, a marked tendency in much of the work on adaptation, as Simone Murray (2012: 186) put it so eloquently, “overwhelming focus[es] on the semiotic richness of adapted texts,” resulting in case studies that are largely formalistic. In other words, a considerable amount of adaptation scholarship has been based on the unproblematized tallying up shared and not shared features of the text and the adaptation, without much theorizing as to why such features are significant or what the absence or inclusion of a particular feature might signify. In such cases adaptation is seen as “a matter of searching two systems of communication for elements of equivalent position in the systems capable of eliciting a signified at a given level of pertinence” (Andrew 1984: 13). Furthermore, although the concept and the practice of intertextuality reside at the heart of adaptation, there is a certain negligence in addressing the practice on the side of the text outside of or before its relationship with its film adaptation. What is often ignored is the profoundly intertextual make-up of the text that has been adapted, with the result that it is treated as a homogenous and stable entity. In such cases, it is as though an opportunity has been missed to tease out all of the complexity of the text in its relation to other texts and indeed whole traditions, in the service of analyzing how it was brought to the screen. Moreover, if adaptation studies are currently being pushed in any direction, it is towards thinking beyond text-to-film dyads. For example, Shannon Wells-Lassagne and Ariane Hudelet have argued for moving adaptation studies “beyond the simple relationship between novel and film, [to] expand [analysis in order] to examine less-studied media” and, more generally, to the current massive “culture of rewriting and re-appropriation” (Wells-Lassagne/Hudelet 2013: 1). Likewise, Hassler-Forest and Nicklas have noted the current interest in “cross-media” adaptations and the need to study “numerous textual relations [such as] novels that have become adapted into films, television series based on literary properties, video games based on cinematic franchises, and so forth” (Hassler-Forest/Nicklas 2015: 2). Similarly, the notion of “transmedial storytelling” has emerged as scholars address the “spreadability” (Jenkins 2006) of stories through complex networks of platforms including literary properties in text form, as well as comic books, TV shows, films, video games, and the list goes on. And finally, Hassler-Forest and Nicklas also argue for the importance of politics in studying adaptation given that, as noted above, “adaptation processes inherently imply hierarchies of taste and cultural capital (Bourdieu 1984), [hence] both the theory and the practice of adaptation are ultimately determined by social relationships that are inherently political” (Hassler-Forest/Nicklas 2015: 2). Likewise, I would wholeheartedly concur with their call to take the effects of globalization, media convergence, and audience participation into consideration as these impact on the political economy of adaptation.

Eat, Pray, Love

173

2 Adaptation, Economy, Intertextuality: Eat, Pray, LovE In my own work on adaption, I have focused on what is being adapted (cf. Goggin 2011) beyond details such as events that drive the plot or inform the setting. My interest lies in, for example, how adaptations of a particular text such as Jane Austen’s ‘Pride and Prejudice’ (cf. Goggin 2007) recount the political and economic context in which they were produced or how film remakes, such as the Ocean’s (cf. Goggin 2010) movies, consistently adapt the circulation of money through institutions such as banks and casinos, as well as the political concerns that coalesce around the economy. While to some this approach may seem to side-step the question of considerations such as the aesthetic properties of a given film, I would argue that adaptations are industrial products created in ‘dream factories’, and that the underlying asset – the text – is equally a consumable that tells the story, however implicitly, of the economy in which it was written. As such, the aesthetics of any adaptation, as well as its pace and various period details reproduced therein, are implicitly and explicitly informed by the rapidity with which money moves and how it circulates in the context in which it was produced. In my reading here of ‘Eat, Pray, Love’ (Gilbert 2010) therefore, I will follow David Harvey who has argued that, “[i]t should be possible to consider how […] the changing experience of space, time and money has formed a distinctive material basis for the rise of distinctive systems of interpretation and representation, as well as opening a path through which the aestheticization of politics might once more reassert itself. If we view culture as that complex of signs and significations that mesh into codes of transmission of social values and meanings, then we can at least begin upon the task of unravelling its complexities under present-day conditions by recognizing that money and commodities are themselves the primary bearers of cultural codes. Since money and commodities are entirely bound up with the circulation of capital, it follows that cultural forms are firmly rooted in the daily experience of money and the commodity […].” (Harvey 1990: 299, my emphasis) Keeping Harvey’s words in mind, I have chosen Gilbert’s ‘Eat, Pray, Love’ not only because of the way it mimics the structures and practices of the market in which it was produced and in which it was intended to make money, but also because of its remarkable relationship to tourism, globalization, and the economics of various industries that trade on narrative. Indeed, my interest in this text was initially piqued by a stack of copies of ‘Eat, Pray, Love’ that greeted me at an airport bookstore with covers announcing the upcoming film adaptation, precisely because the book-cover tie-in to the film starring Julia Roberts points directly to the profit motives behind both text and film.

174

Joyce Goggin

My interest in this text and film is also connected to the relationship of both book and film to the notion of expanding adaptations, which is to say that Gilbert’s “foodie romance,” “confessional memoir,” “culinary adventure,” “gastronomic travelogue,” or “priv lit” narrative as it has been called, is an expanded adaptation itself, even before being made into a movie (Thoma 2014: 109). Indeed, the difficulty of classifying the text either as “travel writing”, “autobiography”, or “self-help fiction” (Thoma 2014: 109), points to the expansiveness of this “woman’s search for [nothing less] than everything across Italy, India and Indonesia” (Gilbert 2010), a journey and a dynamic that finds its sources in the kinds of intertexts that Gilbert’s narrative absorbs and reproduces. Eat, Pray, Love is an expanded and expanding adaptation that incorporates travel and food guides in the ‘Italy’ section; self-help and spiritual guide-books in ‘India’; and popular anthropology reminiscent of National Geographic in ‘Indonesia’, which all find their resolution in classic romance. And finally, I am also interested in how Gilbert’s narrative, unlike a crime drama that could be adapted as a film, TV series, or video game, has taken on an intermedial life of its own in perhaps unexpected ways across film and other narrative properties, as well as travel destinations and lifestyle choices. In what follows then, I will discuss Eat, Pray, Love and how both novel and film maybe be productively read as expanded adaptations with a surprisingly wide reach and serious political consequences, beginning with ‘Italy’.

2.1 Eating Italy: Tourism, Travel Promotion, and Eat, Pray, LovE In an article provocatively entitled “Eat, Pray, Love Turned Me Into a Selfish Bitch”, Carrie Goldberg (2010) has written that “Rome was by far the best part of the movie. Luxurious, beautiful, epicurean, decadent – [every]thing I would want my own European vacation to be.” The kinds of cinematic moments to which Goldberg refers, are written into the ‘Italy’ section of the text, in passages where Gilbert strolls by Villa Borghese as “the sun set[s] gold over St. Peter’s Basilica,” (Gilbert 2006: 60) or whistles by train stops named for “the world’s most famous foods and wines: next stop, Parma … Next stop, Bologna … next stop Montepulciano” (Gilbert 2006: 131). So while these cues throughout the text suggest that the novel would be highly adaptable as a film, in the ‘Italy’ section Gilbert also consciously includes anecdotal information gleaned from guidebooks concerning public monuments, such as the “gorgeous fountain at the Piazza della Repubblica” featuring “bodacious naked nymphs” (Gilbert 2006: 114). “According to my Guidebook”, she writes, “the women who modeled for the nymphs were a pair of sisters, two popular burlesque dancers of their day […]. The sisters lived well into old age, and even as late as the 1920s […] every year, once a year, for

Eat, Pray, Love

175

as long as he lived, the French sculptor who had captured them in marble during their prime would come to Rome and take the sisters out to lunch.” (Gilbert 2006: 114) Where consulting the Michelin is concerned, she is trumped only by her sister who arrives with “five guidebooks”, “has the city pre-mapped in her head” and spends her time in Italy pointing out the “Gothic, or Romanesque, or Byzantine features” of buildings, along with “19-century façade[s]” and “Roman monoliths” (Gilbert 2006: 118–119). In other words, from the outset Gilbert’s text very consciously adapts the travel narrative and guide book genre, while inviting its own cinematic adaptation. But of course, she is also in Italy to eat hence the decidedly foodie tone from Gilbert’s first meal in Rome, which she litotically describes as “nothing much”, consisting of “homemade pasta […] sautéed spinach and garlic […] artichokes”, “fried zucchini blossoms […] (prepared so delicately that the blossoms probably didn’t even notice they weren’t on the vine anymore)”, veal and tiramisu (Gilbert 2006: 45). She likewise plugs specific locations and venues such as Pizzeria di Michele in Naples, reputed to offer “the best pizza in the world”, which reduces her companion “practically to tears” as she has “a metaphysical crisis over” the “sweet tomato sauce that foams up […] when it melts the fresh buffalo mozzarella, and the one sprig of basil in the middle [… that] somehow infuses the entire pizza with herbal radiance” (Gilbert 2006: 104, 105, 106; emphasis in original). Given the explicit references to food and travel guides in the ‘Italy’ section and throughout the rest of the text, it is hardly surprising that ‘Eat, Pray, Love’ was promoted heavily in airports and rapidly spawned its own industry of guided tours, wellness and experience vacations, following Gilbert’s route to self-actualization. Indeed one report on “the intensity of her fanbase” describes women at a reading who said “that this book inspired them to travel”, hence, in David Harvey’s words, both text and film exemplify how, “[…] mass tourism, [… and] films made in spectacular locations, [make] a wide range of simulated or vicarious experiences of what the world contains available to many people.3 The image of places and spaces becomes as open to production, and ephemeral use as any other” (Harvey 1990: 293). The importance of the imagination and its role in opening the world up for various kinds of wealth production based on the commodification of experience and “ephemeral use” has been commented on at length by Arjun Appadurai, who wrote that “[t]he imagination […] is itself a social fact, and is the key component of the new global order” (Appadurai 1996: 5). It expresses itself through the construction of mediascapes that “tend to be image-centered, narrative-based accounts of strips of reality, and what they offer to those who experience and transform them

176

Joyce Goggin is a series of elements (such as characters, plots and textual forms) out of which scripts can be formed of imagined lives, their own, as well as those of others living in other places. These scripts can and do get disaggregated into complex sets of metaphors by which people live […] as they help to constitute narratives of the Other and proto-narratives of possible lives.” (Appadurai 1996: 9)

Polly Pattullo has likewise argued that mass tourism, spurred on by work like Gilbert’s that straddles the line between fiction and fantasy, results in nothing less than “the rewriting of the economic and political geography of the world” (quoted in Carrigan 2011: 9), while according to Michel Picard, global tourism like that promoted by ‘Eat, Pray, Love’ severs the landscape and “the cultural tradition of a society” from their contexts, and facilitates their “serialization” as tourist products (Carrigan 2011: 10). The kinds of stereotypes circulated by industries that promote tourism, and in which ‘Eat, Pray, Love’ abounds, create maps – like the one in her sister’s head – of commodified landscapes for the global tourist, that turn countries into “a stage prop or movie backdrop” (Carrigan 2011: 16). And no one expresses this better than Gilbert herself, who explains that she “noticed that all these countries [she visits] begin with the letter I. A fairly auspicious sign […] on a journey of self-discovery” (Gilbert 2006: 38), and that her gap-year is undertaken “not so much” with the goal of “thoroughly explor[ing] the countries themselves” (Gilbert 2006: 37). No, she explains, “[i]t was more that I wanted to explore thoroughly one aspect of myself set against the backdrop of each country” (Gilbert 2006: 37).

2.2 Self-Help and Derivative Logic Anthony Carrigan is yet another scholar who has remarked that social imaginaries of tourist destinations are partially the product of “creative writing’s relationship with reality”, as typified by Gilbert’s narrative of the narcissistic journey on which she ravenously consumes national cuisines, cultures, and religions (Carrigan 2011: 28). Of equal relevance here is James Clifford’s (1997: 105) observation that the “discursive/imaginary topographies of Western travel are being revealed as systematically gendered, symbolic stagings of self and other”, and I want now to focus on the ‘India’ section, and what I take to be the deep logic of such gendered stagings, through the medium of Gilbert’s creative relationship with reality. Importantly, Gilbert’s self-staging or what she calls her “physics of self-discovery” is supposedly about “stepping off the track of a conventional lifestyle and losing all the embracing comforts that keep so many people on that track” (Gilbert 2006: 125), yet it is clear that this “priv lit” narrative can only be viewed through the lens of constant consumption, undertaken by an author wealthy enough to afford a gap-year in three different countries. Indeed, when she announces

Eat, Pray, Love

177

her plan to friends, those more cynical ask why she doesn’t simply go on a “pilgrimage through the Great Tri-State “I” Triumvirate of Islip, I-95 and Ikea” (Gilbert 2006: 38). Where this last point is concerned, the middle section of ‘Eat, Pray, Love’, set in an Ashram, marks a moment in which Gilbert learns self-love and how to meditate, while her discourse aligns itself with self-help and inspirational literature, a tradition whose roots are long, predominantly American and planted in the rich soil of money. The imprint of economics on American models of the psyche is evident in early self-help books, which Micki McGee (2005), in her major work on self-help and make-over culture, has traced to Benjamin Franklin’s account of the entrepreneurial individual, in his autobiography, as well as in the ‘Poor Richard Almanack’, published serially from 1732 to 1758. Permeated with a colonial version of neo-classical economics, these guides to self-actualization taught that sound sleeping habits and a strict economy would make “a man healthy, wealthy and wise” (Franklin 1855: 13). A century later Samuel Smiles published ‘Self Help’ (1859), a work influenced by Benjamin Franklin that would instruct the British and Americans in the art of ‘Self-Culture’. In Smiles’s work, the road to self-improvement is signposted by chapters such as “Men of Business”, or “Money – Its Use and Abuse”, and paved with advice such as “the rich are the diligent, who can command Time, nature’s stock!” (Smiles 1859: 174, 191, 64). The following century of self-help would get underway with Carnegie’s ‘How to Win Friends and Influence People’ (1936), Napoleon Hill’s ‘Think and Grow Rich’ (1937) and later Claude Bristol’s ‘The Magic of Believing’ (1948), all of which are predicated on the equation of happiness with capital, gained through self-fashioning and motivated by what Harold Bloom described in ‘The American Religion’ as a “religiously mad culture, furiously searching for the spirit” (Bloom 1992: 22). At the same time, the self-help tradition is likewise informed by what Max Weber, also referring to Ben Franklin, saw as the ‘Spirit of Capitalism’, that is, a social order in which spirituality has migrated into capital and become secularized as a doctrine of the self. The self-help tradition as we currently know it was born of Freudian psychoanalysis transplanted into the United States, where it was democratized, and made affordable for a wide audience through its commodification in various forms, including the self-help book. While it is appealing to see the monetization of the human psyche as an entirely American phenomenon, it is important to keep in mind that Freud himself opened ‘Beyond the Pleasure Principle’ (1922), by describing his “model of mental processes” as being structured on a utilitarian equation governed by the “avoidance of pain [and] the production of pleasure” (Freud 1961: 71). As Freud explained, the psychic double-ledger accountancy at the heart of this model introduces a utilitarian “economic point of view” into his work on human consciousness (Freud 1961: 1). In other words, the Freudian model of the psyche was already based on utilitarian accountancy and economic

178

Joyce Goggin

realism, which was then popularized and commodified to fit both the temporal and pecuniary needs of the American public by authors like Hill and Bristol. According to Hill, the most salient of these needs is the development of what he called “money consciousness”, which he proposed as a means of remedying “the six-year economic collapse”, while propounding the importance of that “LACK OF FAITH” that was the root-cause of the great depression (Hill 1937: 36, 37, 61). In the intervening years, our lives have become thoroughly ‘financialized’, to borrow Randy Martin’s term, by which he refers to the on-going process of deregulation that has led to the market’s intrusion into every aspect of our lives, and making productive every possible space for the creation of wealth and value. This includes health care, the liquifaction and actionability of domestic space, subjective profiles managed as asset portfolios and various entertainment industries that thoroughly financialize our leisure time. More recently, Martin (2011) has introduced the notion of derivative logic or a logic of assemblage that structures not just the housing market that crashed in 2008, but virtually every aspect of our daily lives. This hinges on the capacity of “derivatives [to] take all manner of different commodities and connect them through a particular attribute that renders local goods into a global market for various kinds of credit and debt” while it “generates its own economic opportunities and dynamics” (Martin 2011: 158–159). Or, as Gilbert (2006: 37) writes, the idea, is to “somehow create an expansive enough life that [one] could synchronize seemingly incongruous opposites into a worldview that excludes nothing”, and which has created a new dynamic that has grown into an industry that has surpassed the 450 million dollar mark. And this financial success mimics the logic of the derivative in one other important regard: just as tranching and recombining assets was intended to spread risk and therefore avoid losses, Gilbert’s technique of combining “seemingly incongruous opposites” and narrative genres into one makeover story is, however consciously or unconsciously, a technique to cover several markets and avoid the risk of failure. So while earlier self-help reproduced the logic of a market predicated on monetary realism, current self-help reproduces and coaches the kind of superficial yet complex market thinking in and through which we function as subjects. Hence Caroline Myss, who also has constant recourse to monetary metaphors to describe the contemporary psyche, writes that in “a relationship […] contributions and benefits need not be spelled out or exactly equal, but it is important that each person feels he or she is receiving good value in exchange for what is offered” from one’s “grace bank account” (Myss 1998: 16, 64) And through this discourse of spiritual stock trading emerges what she calls “woundology” whereby we share our wounds as short-hand for depth of character in this thoroughly financialized world. To remedy this she, like Gilbert who spends 445 pages sharing her wounds, proposes a new

Eat, Pray, Love

179

“spirituality that is more universalist in orientation” and “open to a staggering range of spiritual traditions and practices” which she also proceeds to tranche like so many toxic assets (Myss 1998: 86). Moreover, just as the self-help texts of the 1930s and 1940s encouraged investors to re-engage with the market after the depression, Gilbert, Myss, and many others including everyone from games scholar Jasper Juul (2013) and Prince Andrew, now teach the value of failure as an opportunity to begin anew.4 Put differently, Gilbert’s journey in which she learns to get back in the water after the disaster is analogous to the “contemporary processes of globalization [that] demonstrate that capitalism, in its cycles of creative destruction and resurrection, has again reinvented itself” (Lee/Lipuma 2002: 209). In the present case, Gilbert’s reinvented, post personal crash self gave rise to a diversified derivative package that includes a new fictional form, a movie, and several other outputs from a crowd-sourced home in Bali, to a new lover, and a whole range of ‘Eat, Pray, Love’ products sold globally such as candles, jewelry, prayer mats, perfumes, and other beauty products, herbal teas, and so on.

2.3 Anthropology If the first two sections of ‘Eat, Pray, Love’ are imprinted with the discourses of travel and self-help, I would argue that the third section in Indonesia reads like armchair anthropology, particularly in Gilbert’s lengthy observations concerning the complexity of Balinese social structures which recall Clifford Geertz’s famous essay on cockfighting in Bali. Importantly, however, in James Clifford’s work on travel, he writes that in representations of tourists and anthropologists alike “the means of transport is largely erased” because modes of transport “suggest systematic prior and ongoing contacts and commerce with exterior places and forces that are not part of […] ‘being there’” (Clifford 1997: 99), and this most certainly applies to ‘Eat, Pray, Love’ wherein Elizabeth Gilbert and Julia Roberts are only rarely and briefly represented in transit.5 Rather, the text and the camera always find the heroine at her destination, framed in spectacular establishing shots, having done none of the heavy lifting, and ready to perform the affective and emotive labor of the spiritual make-over. What is likewise part of Gilbert’s armchair anthropology, is the way in which she uses the places and people with whom she comes in contact. In every setting, she either receives guests from home with whom she can discuss her emotional wounds, or meets someone, like Richard from Texas with whom she connects at the Ashram in India, while she treats the locals as ‘Squantos’ or local guides to the culture, that is, as “disconcertingly hybrid ‘native[s]’ met at the ends of the earth; strangely familiar and different […] complex individuals routinely made to speak for ‘cultural’ knowledge” (Clifford 1997: 97). In this regard, it is also not without significance

180

Joyce Goggin

that the speech of her various Squantos is rendered in heavily accented and ungrammatical English such as Ketut Liyer’s story in which he explains “it is nine generations that my family is a medicine man. […] They see I have beautiful, and I have intelligent” (Gilbert 2006, 307). And perhaps somewhat more offensively still, in one scene in the ‘Pray’ section of the film, the locals are rendered as cardboard cut-outs and have become quite literally part of the wraparound backdrop against which Gilbert projects her spiritual awakening, and who, in the words of Arjun Appadurai, “help to constitute narratives of the ‘other’ and proto-narratives of possible lives, fantasies which could become prolegomena to the desire for acquisition and movement” (Appadurai 1996: 36).

Figure 8.1 Julia Roberts emotes in front of cardboard cut-out locals in Eat, Pray, Love.

3 Eat, Pray, LovE and Expanded Adaptation To conclude with the notion of expanded adaptations with which I began, Pamela Thoma has remarked that the capacity for absorption and assemblage that distinguishes much chick lit, and ‘Eat, Pray, Love’ in particular, makes it “readily available for revision and re-mediation, with migrations and collisions that have transported it […] into film” and various other platforms precisely because it is “itself a hybrid and fictionalized form of biography, that recombines fantasy and realism for further variation of the makeover narrative that confounds strict medium formulation.” (Thoma 2014: 114) While I have been arguing this point in terms of textuality and the book’s relation to other textual genres such as travel and spirituality guides, and the cultures and “worldviews” that Gilbert wants to “incorporate into” her

Eat, Pray, Love

181

own, she also constantly hints that her book could be a movie (Gilbert 2006: 302).6 For example, she provides endless cues to visualizing the text cinematically, such as the pizza scene in which she describes the herbal radiance of a sprig of basil as being analogous to “the way one shimmering movie star […] brings a contact high of glamour to everyone around her” (Gilbert 2006: 106). Likewise, she describes her own “greedy madness” for travel, as equivalent to “fantasizing about having sex with your favorite movie star while you’re having sex with your other favorite movie star”, and so I now want to conclude by approaching the question of how Gilbert’s text has had sex with the film industry (Gilbert 2006: 102). To use Anne Friedberg’s terms, I would argue that ‘Eat, Pray, Love’ is an example of “the mobilized gaze” that lights upon static displays such as tourist destinations and shop windows, and is “characterized by cultural activities that involve walking and travel” (Friedberg 1993: 2). With the advent of cinema, the gaze becomes virtual and serves as a ‘received perception’, enjoyed in the early years of film in the form of newsreels and travel documentaries, such as the one recorded by Thomas Mann as his first experience of film. Life, he writes, “flit[ed] across the screen […] a thrilling drama of love and death […] silently reeled off; the scenes, laid at the court of an oriental despot […] or a young Moroccan woman, in a costume of striped silk, was suddenly brought so close to the camera as to be life-sized; while with one hand she waved to the audience, who stared, taken aback, into the face of the charming apparition. Constructed, […] to cater to the innermost desires of an on-looking international civilization […].” (Mann 1927: 316–317)

Figure 8.2 Julia Roberts with one of her squantos in Eat, Pray, Love.

182

Joyce Goggin

In the present context, Mann’s description of his initiation to the cinema is particularly prescient. The travel documentary form, like ‘Eat, Pray, Love’ and its film adaptation, are created to foreground both the magic of the cinema in its capacity to collapse time and space thereby transporting the viewer to distant lands, as well as the increased mobility made possible by the numerous economic and technological affordances of western modernity. I would argue then, that both the narrative and the film of ‘Eat, Pray, Love’ are constructed around the aesthetic logic of the shop window display and the travel agent, built to feed audiences with the logic of the current capitalist paradigm that encourages frequent travel, and self-actualization in lieu of a social safety net. Indeed, Gilbert’s spiritual pilgrimage is undertaken by a subject whose spirituality is rooted in capital, and whose project contradicts both the author’s eco-message as well as her claim to being “brave enough to leave behind everything familiar and comforting”. The finished product is a filmed version of spiritual tourism replete with “over four hundred product tie-ins”, which has aided in increasing tourist traffic to Gilbert’s destinations and which prompts consumers to enjoy “cultural artifacts and practices of the East” and the “cultural capital that defines the neoliberal global cosmopolitan” (Thoma 2014: 117). To return to the politics of adaption, I have suggested along with Appadurai that products of the imagination like Gilbert’s are key in the construction of the new global order, and with Pattullo that the kind of self-writing combined with travel narrative contained in ‘Eat, Pray, Love’ is a key factor in the economic and political rewriting of the world. In other words, this kind of writing is adapted not just in other dematerialized entertainment products: it also impacts on the very geography of our life worlds and particularly on the political economy of countries that have rebranded and reshaped themselves in line with tourists’ desires and expectations. On a smaller, more granular scale, Gilbert’s work has been adapted by countless women the world over who have, to varying degrees, turned their lives into adaptations of ‘Eat, Pray, Love’ by travelling to her destinations and following the Gilbert tour, or by consuming the various self-actualization and spiritual growth products that have been adapted from her text. Moreover, since its publication, ‘Eat, Pray, Love’ has spawned a genre of travel-for-spiritual-growth-and-happiness narratives, such as Cheryl Strayed’s 2012 memoir ‘Wild: From Lost to Found on the Pacific Crest Trail’, also adapted as a movie in 2014. This narrative/film dyad offers a make-over model for women who cannot afford to take a gap-year to travel the world, targeting a more down-market sector of female consumers, that might be able to spend a month hiking the Pacific Crest Trail. And, inspired by Gilbert’s experiment in self-actualization, Gretchen Rubin has published ‘The Happiness Project’, which recounts her year of self-fashioning at home. The text is offered as a guide to those who can’t go anywhere, and contains a project calendar to ensured sustained personal growth and dedication without travel. More generally, in the face of increasing precarity in the

Eat, Pray, Love

183

work place, financial bubbles that have put many out of their homes, and looming ecological disaster, Gilbert’s narrative and others like it have been adapted in any variety of happiness culture products, songs, films and so on that are instructing us all in the discipline of self-actualization whereby failure is currently being touted as the new success.

Notes 1. Notable examples include Chatman (1980), Branigan (1992), and Corrigan (1999). 2. On this point see “Adaptation or Something Like It: Karl Reiz’s gambler” (Goggin 2013), in which I discuss an adaptation that blatantly misquotes the tutor text in the scant references it makes to Dostoyevsky’s ‘The Gambler’, and does not take much more from the text than the title. 3. See: http://flavorwire.com/453038/the-skeptics-guide-to-elizabeth-gilbert [last accessed: 01 March 2016]. 4. See: “One must teach children to fail, says Prince Andrew”, online: http:// www.theweek.co.uk/uk-news/57916/we-must-teach-children-fail-saysprince-andrew#ixzz3677MsllA [last accessed: 01 March 2016]. 5. One exception is Gilbert’s voyage to the Ashram in India. In the text Gilbert compares her arrival there to introducing a new chicken to the coop in the middle of the night so that it simply appears to the other chickens in the morning, as though it had always been there. Rather than masking her arrival as in the text, the film renders this event in an extended scene of travel and contains many shots of beggars thrusting their hands through the window of the taxi that takes Roberts to the Ashram, perhaps reassuringly playing up to received notions of Indian poverty and stereotypes of street beggars in countries less privileged than a ‘safe’ country like Gilbert’s native USA. 6. Cf. Cohen: “[T]he demonstration of the cinematic quality of the modern novel has been available ever since the first reader of Ulysses noted the montage technique of ‘The Wandering Rocks’ or since Dos Passos included the ‘News-reel’and ‘Camera Eye’ sections in U.S.A. In other words, certain modern novels proclaim themselves cinematic” (Cohen 1979: 2).

Bibliography Andrew, D. (1980). “The Well-Worn Muse: Adaptation in Film History and Theory.” In: Conger, S.M./Welsch, J.R. (eds.). Narrative Strategies: Original Essays in Film and Prose Fiction. Macomb: Western Illinois University, pp. 9–17. Appadurai, A. (1996). “Disjuncture and Difference”. In: Appadurai, A. (ed.). Modernity at Large: Cultural Dimensions of Globalization. Minneapolis: Minnesota UP, pp. 27–47. Aristotle (1982). Poetics. New York: W.W. Norton & Company, Inc. Baker, B. (2009). “Gallivanting Around the Worlds. Bond, the Gaze and Mobility.” In: Lindner, C. (ed.). Revisioning 007. James Bond and ‘Casino Royale’. London: Wallflower Press, pp. 144–158. Bloom, H. (1992). The American Religion: The Emergence of the Post-Christian Nation. New York/London: Simon & Schuster.

184

Joyce Goggin

Bluestone, G. (1957). Novels into Film: The Metamorphosis of Fiction into Cinema. Berkeley/Los Angeles/London: University of California Press. Bourdieu, P. (1984). Distinction: A Social Critique of the Judgment of Taste. Cambridge: Harvard University Press. Branigan, E. (1992). “Narration.” In: Branigan, E. (ed.). Narrative Comprehension and Film. London: Routledge, pp. 63–85. Bruhn, J./Gjelsvik, A./Frisvold Hanssen, E. (eds.) (2013). Adaptation Studies: New Challenges, New Directions. London/New York: Bloomsbury. Carnegie, D. (1936). How to Win Friends and Influence People. New York: Simon & Schuster. Carrigan, A. (2011). Postcolonial Tourism: Literature, Culture, and Environment. New York/London: Routledge. Cartmell, D./Whelehan, I. (eds.) (1999). Adaptations: From Text to Screen, Screen to Text. New York/London: Routledge. Chatman, S. (1980). “What Novels Can Do That Films Can’t (and Vice Versa).” Critical Inquiry 7(1), pp. 121–140. Clifford, J. (1997). “Traveling Cultures.” In: Clifford, J. (ed.). Routes: Travel and Translation in the Late Twentieth Century. Cambridge: Harvard UP, pp. 17–39. Cohen, K. (1979). Film and Fiction: The Dynamics of Exchange. New Haven/ London: Yale UP. Corrigan, T. (1999). Film and Literature: An Introduction and Reader. Upper Saddle River: Prentice Hall. Dostoyevsky, F. (1996). The Gambler. New York: Dover. Elliott, K. (2003). Rethinking the Novel/Film Debate. Cambridge: Cambridge University Press. Franklin, B. (1855). Early to Bed, and Early to Rise, Makes a Man Healthy, Wealthy, and Wise or Early Rising. London: Abel & Sons. Freud, S. (1961). Beyond the Pleasure Principle. New York/London: W.W. Norton & Company. Friedberg, A. (1993). Window Shopping: Cinema and the Postmodern. Berkeley/Los Angeles/London: University of California Press. Geertz, C. (1972). “Deep Play: Notes on the Balinese Cockfight.” In: Geertz, C. (ed.). The Interpretation of Cultures. New York: Basic Books, pp. 412–453. Gilbert, E. (2006). Eat, Pray, Love. One Woman’s Search for Everything Across Italy, India and Indonesia. New York: Penguin. Goggin, J. (2013). “Adaptation or Something Like It: Karl Reiz’s Gambler”. In: Raw, L. (ed.). The Silk Road of Adaptation. Cambridge: Cambridge Scholars Press, pp. 64–78. Goggin, J. (2010). “From Remake to Sequel: Ocean’s Eleven and Ocean’s Twelve.” In: Jess-Coke, C./Verevis, C. (eds.). Second Takes: Approaches to the Film Sequel. New York: State University of New York Press, pp. 105–121. Goggin, J. (2007). “Jane Austen Reloaded: Portraits and Adaptations.” Persuasions On-Line. Online: http://www.jasna.org/persuasions/index.html [last accessed: 1 March 2016]. Goggin, J. (2013). “Like Pocahontas on Drugs: Avatar and Adaptation”. Interfaces: Image, Texte, Langage 34, pp. 103–113. (Special issue: Expanding Adaptations, edited by Ariane Hudelet and Shannon Wells-Lassagne). Goggin, J. (2011). “Qu’est qu’on réadapte? Ocean’s Eleven et l’esthétique de la finance”. In: Hudelet, A./Wells-Lassagne, S. (eds.). De la page blanche aux salles obscures: Adaptation et réadaption dans le monde Anglophone. Rennes: Presses Universitaires de Rennes, pp. 49–59.

Eat, Pray, Love

185

Goldberg, C. “How Eat, Pray, Love Turned Me Into a Selfish Bitch”. Online: http:// www.jewcy.com/jewish-arts-and-culture/how_eat_pray_love_turned_me_selfish_ bitch#sthash.vHYUByL1.dpuf [last accessed: 1 March 2016]. Harvey, D. (1990). “Time Space Compression and the Postmodern Condition”. In: Harvey, D. (ed.). The Condition of Postmodernity. Oxford: Blackwell, pp. 284–308. Hassler-Forest, D./Nicklas, P. (eds.) (2015). The Politics of Adaptation: Media Convergences and Ideology. London: Palgrave. Hill, N. (1937). Think and Grow Rich. Meriden: The Ralston Society. Jenkins, H./Ford, S./Green, J. (2013). Spreadable Media: Creating Value and Meaning in a Networked Culture. New York/London: New York UP. Juul, J. (2013). The Art of Failure: An Essay on the Pain of Playing Video Games. Cambridge: MIT Press. Lee, B./LiPuma, E. (2002). “Cultures of Circulation: The Imaginations of Modernity”. Public Culture 14(1), pp. 191–213. Leitch, T. (2008). “Adaptation Studies at a Crossroads.” Adaptation 1(1), pp. 63–77. Mann, T. (1927). The Magic Mountain. New York: Alfred A. Knopf. Martin, R. (2011). “Taking an Administrative Turn: Derivative Logics for a Recharged Humanities”. Representations 116 (1), pp. 156–176. McFarlane, B. (1996). “Backgrounds, Issues, and a New Agenda”. In: McFarlane, B. (ed.). Novel to Film: An Introduction to the Theory of Adaptation. Oxford: Clarendon Press, pp. 3–30. McGee, M. (2005). Self-Help, Inc.: Makeover Culture in American Life. Oxford: Oxford University Press. Murray, S. (2012). The Adaptation Industry: The Cultural Economy of Contemporary Literary Adaptation. New York/London: Routledge. Myss, C. (1998). Why Some People Don’t Heal and How They Can: A Practical Programme for Healing Body, Mind and Spirit. London: Bantam Books. Rozario, K. (2007). “Introduction: The Golden Age as Catastrophe.” The Culture of Calamity: Disaster and the Making of Modern America. Chicago: University of Chicago Press, pp. 1–9. Rubin, G. (2009). The Happiness Project: Or, Why I Spent a Year Trying to Sing in the Morning, Clean my Closets, Fight Right, Read Aristotle, and Generally Have More Fun. New York: Harper Collins Publishers. Smiles, R. (1859). Self-Help, National and Individual. Boston: Ticknor and Fields. Strayed, C. (2012). Wild: From Lost to Found on the Pacific Crest Trail. New York: Vintage. Thoma, P. (2014). “What Julia Knew: Cooking, Writing, and Other Forms of Domestic Labor in the Recession-Era Chick Flick”. In: Negra, D./Tasker, Y. (eds.). Gendering the Recession: Media and Culture in an Age of Austerity. Durham: Duke University Press, pp. 165–203. Wells-Lassagne, S./Hudelet, A. (2013). “Introduction”. Interfaces: Image, Texte, Langage 34, pp. 1–9 (Special issue: Expanding Adaptations, edited by Ariane Hudelet and Shannon Wells-Lassagne).

Filmography Clueless (1995). Amy Heckerling. Paramount Pictures. USA. Eat, Pray, Love (2010). Ryan Murphy. Columbia Pictures Industries. Italy/India/ Indonesia/USA.

186

Joyce Goggin

Ocean’s Eleven (2001). Steven Soderbergh. Warner Bros. USA. Ocean’s Thirteen (2007). Steven Soderbergh. Warner Bros. USA. Ocean’s Twelve (2004). Steven Soderbergh. Warner Bros. USA. Romeo + Juliet (1996). Baz Luhrman. Bazmark Films/Twentieth Century Fox Film Corporation. USA. Vanity Fair (2004). Mira Nair. Focus Features/Tempesta Films/Granada Film Productions/Inside Track Films/Mirabai Films/Cine Mosaic. USA/UK/India. Wild (2015). Jean-Marc Vallée. Fox Searchlight Pictures/Pacific Standard. USA.

9

Conclusion Film Text Analysis – A New Beginning? Janina Wildfeuer & John A. Bateman

1 Filmic Understanding in the Spotlight One of the most fundamental questions that has been asked within film theory over the course of time, highlighted again and again by researchers from all kinds of disciplines connected to film studies, is the following: How do we understand films? According to David Bordwell, this question refers not only to the basic level of interpretation and explanation of the film’s so-called narrative or textual logic but also includes meta-levels of interpretation going beyond perceptual and cognitive processes during the reception of a film (cf. Bordwell 2011). In Bordwell’s conception, the question can be approached beneficially by drawing into film theory the notion of common sense. This notion is seen as offering a means for bringing together the various levels of description and analysis necessary for understanding ‘film understanding’, and allows all aspects of filmic meaning to receive closer consideration, including not only the technical details of filmic composition and their meaning-making qualities, but also the needed contextual and socio-cultural information and, for instance, the role of the spectator in all of these processes. Bordwell takes this as an effective means of working beyond any idea that filmic expression (and, indeed, visual perception in general) is arbitrary. The idea of ‘arbitrariness’ still lingers in many accounts due to straightforward applications of the notions of ‘codes’ in the spirit of Saussure that we discussed and critiqued in the introduction to this volume. In psychology and studies of perception, it has now been established that processes of interpretation are best seen as working in both directions, ‘empirically’ on the basis of data and sensory input, and ‘rationally’ on the basis of interpretative schemes, background knowledge, and so on. This kind of interplay raises a variety of problems for older notions of codes, but are part of the regular process of textual interpretation presumed in more current dynamic accounts (cf., e.g., Coëgnarts/Kravanja 2015a; Reinhard/Olson 2016). Nevertheless, various challenges are also raised around the question of the appropriate contents and organizations for the schemes and background knowledge to be assumed when making sense of ‘data’. Bringing in common sense at this point usefully constrains just what that information might be and anchors just what is to be considered ‘obvious’ for film recipients as

188

Janina Wildfeuer & John A. Bateman

the basis for their interpretative efforts. Such enrichments of the information available for ‘pre-structuring’ perception can now be found in many accounts because the top-down processing of perceptual cues is evidently considerable. The extent to which this applies in the specific case of film is, furthermore, now a very active area of research (cf. Loschky et al. 2015; Kluss et al. 2016). It is then natural that methods from psychology and the neurosciences find increasing application when seeking sources of evidence for accounts of film understanding. One result of such considerations has been Bordwell’s reappraisal of his older view of the operation of narrative. Bordwell (1985), for example, saw the construction of the fabula, i.e., the temporally and causally sequenced events of the story, as one example of extensive inference at work. As Bordwell (2011) explains, however, there turns out to be little empirical evidence that actual viewers construct such a complete representation during film reception and so the role of such construction for the overall model of film understanding needed to be relativized. Inference only appears to work within a very narrow timeframe, naturally yielding a more idiosyncratic construction in memory of what is taken to have occurred in a film. A significant part of this enterprise of understanding is then suggested to lie in the realm of ‘folk psychology’, i.e., common-sense views of what people do and why, how they react, and so on. This retains the general idea that the recognition of what is going on in films depends on everyday perceptual abilities, but extends just what is to be considered as belonging to this notion of the ‘everyday’. Consequently: “Recognizing that it [common sense] is in play in narrative comprehension makes it something we need to analyze. We can understand filmic understanding better if we recognize what’s intuitively obvious, and then go on to ask what in the film, and in our psychological and social makeup, makes something obvious. And those factors may not be obvious in themselves. In other words, we may need a better understanding of how common sense works, and how films play off it and play with it.” (Bordwell 2011) There is little doubt that this is the case: the filling in of perceptual hypotheses on the basis of what we know about the world and others appears irrefutable. For this volume, however, where we need to engage more with this line of discussion is the extent to which it offers a sufficient account of filmic practice and interpretation – that is, might we need more than just common sense and, if so, what? We relate this to our considerations of textuality as follows. Bordwell’s discussion of some of the effects of common sense draws attention to many of the psychological principles that appear to be at work in film interpretation. The idea is that since these principles are basic to our cognitive

Conclusion

189

makeup, they do not require separate treatment as part of a theory of film; film understanding should follow ‘directly’ by what we consequently see as common sense. Such principles include long studied phenomena such as the primacy effect, in which what is encountered first gives rise to dispositions for interpreting what follows, the recency effect, where what comes last exhibits a strong shaping influence on any organization given to a sequence of events as a whole, and confirmation bias, where supporting evidence receives more weight than potentially far more significant counter-evidence. Cognitive mechanisms of this kind can often be seen to be at work in film and, indeed, in most communicative forms, language included. There is, however, very little expectation in studies of verbal communication that its forms and practices can be reduced entirely to basic cognitive operations of this kind. As shown in accounts of syntax, semantics, pragmatics, discourse, genre, and much more, there are important additional levels of organization and abstraction to be considered, and these are, in fact, all central phenomena of textuality. It is of course exceedingly unlikely that any of these linguistic levels of abstraction would work ‘against’, or in opposition to, the supporting mechanisms of neurocognitive processing. But those mechanisms alone only go so far in accounting for textual practice. The proposals for textuality that we have made in this volume should therefore be seen, on the one hand, as building on (or within) neurocognitive mechanisms and, on the other, as characterizing more precisely the radically over-specified nature of filmic cues for guiding interpretations. These patterns of over-specification are what constructs textuality, allowing any cues mobilized to function more effectively as guidelines and signposts within any particular sociohistorically-conditioned stage of development of the medium. Bordwell situates these concerns within his program for a poetics of cinema (e.g., Bordwell 2008). It is then unsurprising that several of the proposals he makes there are more reminiscent of textuality than of perceptual processing. As one example, Bordwell sets out the ‘rules’ for forking path narratives, a narrative-style that has gained considerably in popularity since such prominent mainstream examples as Sliding Doors (Howitt 1998) or Run Lola Run (Tykwer 1998). Bordwell’s rules include “forking paths are linear”, “the fork is signposted”, “forking paths intersect sooner or later”, “forking paths are unified by traditional cohesive devices” (Bordwell 2008: 174–179). All of these rules would be straightforward to describe as textual strategies, but are only indirectly related to the cognitive principles that support them (and many other potential ‘rules’). Characterizing such films in terms of their organized syndromes of textual cues is then a necessary and useful step, particularly considering how this may interact with the narrow temporal frame for inferences also predicted by many formal dynamic models of discourse and text interpretation, where notions of strong ‘locality’ play an important role (cf. Wildfeuer 2014). Here, therefore, the explicit acknowledgment of the text as the locus of application and manifestation

190

Janina Wildfeuer & John A. Bateman

of sociohistorically-situated practices articulates a further resource for integrating input of the various kinds necessary for understanding films, their workings, and their positions within a broader spectrum of possible expository styles. Textual organization in this sense then offers a clear analytical approach that bridges the gap between fine-grained levels of description and higher-order levels of interpretation in film analysis – just as the introduction to this volume motivated. The textual perspective proposed then follows not only from the developments in film theory and film semiotics that we outlined in the introduction but also, and in particular, invites incorporation of, and productive engagement with, recent achievements and new insights from a variety of disciplines connected to film analysis, including contemporary semiotics and multimodal analysis, sound and visual studies, media and cultural studies, recent accounts in text and discourse linguistics, neurocognitive approaches, as well as literary and adaptation studies. These starting points all find places in the approaches discussed by the contributions to this book. Taken together; therefore, they illustrate re-orientations to the particular notion of text as an analytical basis for exploring film interpretation that is capable of addressing broad research issues of general relevance for film studies from theoretical, methodological, as well as analytical perspectives. As the scholars working in these disciplines show in their individual chapters, the insights achieved all presuppose definitions of ‘text’, or the object under study, that go beyond the traditional understandings of ‘text’ found in film theory, referring instead more to contemporary, semiotically or cognitively detailed, and reception-oriented investigations. In short, the analyses all investigate film as textual artifacts according to their possibilities for the design, representation, analysis, interpretation, and evaluation of factual and fictional texts.

2 Film Text Analysis in This Book … We have suggested then that the text analytical perspective on film offers a methodological mediating ground for bringing together a variety of styles of analysis, united by their orientation to the technical details of films and the orchestrated combination of textual cues. In other words, the approach seeks to correlate analytic perspectives, theoretical as well as empirical, by drawing out more clearly and systematically not only just what can be considered to be ‘obvious’ when engaging with film, i.e., what is in any film under investigation, but also how such properties are made to cohere for their guidance of film interpretation. The individual analyses presented illustrate this, albeit in quite diverse ways, thus presenting a broad spectrum of accounts dealing with the analysis of filmic text. It will be beneficial, therefore, to bring the book to a close with an explicit elaboration of how we situate these individual chapters with respect to our overall orientation towards film text analysis, setting out specifically our interpretation

Conclusion

191

of their positions before opening up the discussion to conclude with further directions for research and opportunities for interconnections. Making the positions adopted more explicit in terms of textuality will serve well to emphasize just how this perspective now constitutes a promising area for all kinds of analyses and interpretations of film as one of the most complex, and at the same time broadly comprehensible, types of text currently in existence. Briefly to recap: our characterization of textuality has focused on the role of text as making available a systematically interrelated structuring of ‘cues’. Such structures are seen as having specific properties of their own that can be beneficially utilized for driving film analysis and explanation. Variations on this theme are at work in each of the chapters of the volume, which therefore share a common aim to go beyond the “very general, abstract, socially and individually revealing interpretations of literary (which now includes filmic) works” which have come to dominate much of film research (cf. discussion in Bordwell 2004; Bateman/Schmidt 2012: 23). In contrast, each individual approach has focused on specifically identifiable textual qualities of audio-visual filmic text by pursuing detailed patterns manifest in any filmic text considered. The first three chapters of the book, for example, focused explicitly on individual technical or material criteria of rather diverse kinds: Theo van Leeuwen and Morten Boeriis focused on lighting as a semiotic mode, and hence as one specific meaning-making resource in the audio-visual synthesis; Martine Huvenne addressed the role of sound as a meaning-making device, particularly as relating to the construction of ‘space’; and Wolfgang Wildgen showed some of the interplay between our embodied ‘understanding’ of physical action and movement and its use for constructing events filmically for expository purposes. All three, therefore, show textual qualities, of lighting, of sound and action, that are deployed for guiding interpretation drawing on, but going beyond basic everyday experience. Attention then moved to characterizations of how larger textual units can be constructed from the material distinctions present, drawing on several frameworks that explicitly engage with ‘textual’ organizations of various kinds. Neil Cohn’s chapter demonstrates that general principles of cognitively-motivated narrative structures can be successfully and textually analyzed, suggesting significant further empirical predictions and psychological verification. Similar considerations of broader ‘textual’ organizations are addressed in Janina Wildfeuer’s chapter, in which more concrete distinctions of the dynamic notion of text articulated in the context of multimodal discourse analysis are developed with the aim of describing patterns of multimodal meaning-making in terms of formally organized, but inferentially-driven, discourse structure. John Bateman then characterized various kinds of cues exhibited by cases of media combinations and the phenomena of intermediality in order to explore how blending might help capture their larger-scale effects on textual interpretation: in this case, the

192

Janina Wildfeuer & John A. Bateman

cues function as guides concerning both which ‘spaces’ are to be blended and which components of those spaces need to be placed in correspondence. Finally, Joyce Goggin’s chapter on the filmic adaptation of the novel Eat, Pray, Love represents one account of broader interpretations of societal patterns on a higher, more abstract level which fruitfully combine with the descriptions of technical and semiotic details anchored in the text as also presented in the other chapters. In sum, the book has brought together analyses at the levels of description set out, starting with basic textual patterns of individual filmic aspects such as lighting and sound as well as grammatical and dynamic narrative structures. The more pragmatically-oriented discussions in the latter chapters of the book opened up the perspective in the direction of broader aspects of textuality, such as blending and the film’s general ability to adapt and adopt from other media. The diverse analyses provided thus not only ask how filmic interpretation takes place on the basis of technical details of the film and its material, but also show some of the interpretive possibilities made available by inherent, significant patterns that may be isolated in the text. Such patterns then also shed light on interpretations that not only refer to the semantic meaning of filmic details but also contribute to questions of approaches in literary theory, cultural studies, as well as cognitive film theory. All the chapters of the book, therefore, support the suggested approach of achieving more systematic interpretative work on the basis of an organization and pre-structuring of filmic devices as well as of connections between these devices and their sources of interpretation. This corresponds closely with the general idea of textuality as an approach focusing on precise description and analytical tools for revealing and tracking dynamically unfolding meaningful structures as set out in the volume’s introduction and reiterated above. In this regard, the approaches demonstrate that one of the original, innovative ideas of film theory established and followed over the years, the idea of film text analysis, should still, and again, be seen as a useful and beneficial concept in and for film theory and analysis. The approaches were presented in a specific sequence progressively building with respect to their complexity and abstractness. On the one hand, therefore, the book pursues the task of constructing a comprehensive approach to the issues summarized under the notion of common-sense film analysis; on the other hand (but at the same time), it also goes beyond this notion by including insights and results from contemporary text analytical research that progressively add levels of abstraction essential for more exhaustive approaches to film analysis. As illustrated throughout the book, such levels help make obvious the texture and structure of filmic text as well as the meaningful relationships holding between its contributing components. It is consequently “the general notion of textuality itself [which] then gives us precisely the robust semiotic foundation necessary for approaching the specifics of film” (Bateman/Schmidt 2012: 291).

Conclusion

193

We have explored this approach in considerable detail elsewhere (cf. Bateman 2013; Tseng 2013; Wildfeuer 2014). In this work, we move beyond any particular forms of textuality, such as for example, ‘narrative’, in order to address textual patterns as intrinsic aspects of all discourses; i.e., they constitute a central quality of textuality as such. This allows us to see narrative as just one form of instantiation of more general discursive phenomena. The shift of attention from narrative as a rather specific aspect of film analysis to the more general idea of filmic text can be seen as substantially complementing the commonsensical analysis of narrative comprehension suggested by Bordwell; it also shows why separate constructions of, for example, a fabula (see above) need not arise as this is not a necessary component of dynamic text interpretation. Thus, even though the individual chapters also deal in depth with questions concerning, for example, story and character development or dynamic patterns of narrative structure or grammar, they focus additionally on explicitly textual qualities of the filmic text. These hold independently of, and in addition to, narrative concerns. The common focus of the contributions in this regard is then to be situated in the concrete analyses offered of either technical and semiotic details or broader discursive or intertextual patterns of texts which in turn support interpretations of the meaning-making patterns and cues involved. This provides for a comprehensive interpretation of the filmic examples presented in each chapter not only in terms of narration but more in terms of their working as texts.

3 … and as a New Beginning Film text analysis in the context we have given it here, therefore, denotes a combined description and examination of both the technical details at lower levels of analysis as well as of broader interpretations of societal patterns of the film at higher levels. As a consequence, the analytical results of this combined perspective, all concerning the textuality of film, enrich the analytic toolset available in the broad context of film theory so as to further drive research in the area. Moreover, by productively transgressing the borders of various disciplines, the chapters of the volume also contribute to the potential re-connection and interlocking of film studies, literary science, linguistics, cognitive science, and cultural studies, as well as many other disciplines dealing with audio-visual narratives. This re-connection has been enacted throughout the volume in terms of the mediating function of the ‘filmic text’, suggesting how it is capable of combining inputs from many disciplines and areas within film theory, and spanning issues ranging from film aesthetics and poetics to empirical psychological and neuroscience approaches and cognitivism. The results suggest that such a re-connection and linkage offers a fruitful basis for further research in the broader context of film studies as such. As we argued in the introduction to the volume, however, making the most of

194

Janina Wildfeuer & John A. Bateman

this basis requires that we draw on those developments within linguistics and semiotics that have in recent times redefined the notion of text. The chapters in the book take several steps towards this in their respective illustrations of the benefits of explicitly approaching the complex artifact of film by committing to a degree of analytical precision suitable for supporting methodological, empirical verification. This includes not only the particular understanding of text as a radically multimodal and dynamic inferential process (see above and the introduction) but also the close attention paid to fundamental levels of filmic comprehension operating with the dimensions of materiality and their embodied perception. This latter has become particularly important for film studies, as exhibited in growing interest in new approaches to filmic spectatorship (cf. Reinhard/Olson 2016), as well as to the embodiment of mind, meaning, and thought (cf. Anderson 1996; Grodal 2009; Coëgnarts/Kravanja 2015a). Filmic spectatorship, for example, has been the subject of many exploratory theoretical applications of psychoanalytic, cognitive, and affective theories to film pursuing questions of reception and understanding (cf., e.g., Mulvey 1975; Bordwell 1989; Staiger 1992; Plantinga 2009; Shimamura 2013). This is now being taken further within a “fledgling body of research on film perception and comprehension, which builds largely on studies of reading comprehension […] and scene perception” (Loschky et al. 2015: 2). Film spectatorship under these terms can be seen as “the process of engaging with a film text” in which it is the particular text and its material which offer audio and visual stimuli for the construction of meaning (Reinhard/Olson 2016: 2). While based on earlier socio-cultural and phenomenological models of spectatorship, newer accounts increasingly take a strong neurophysiological perspective, including dimensions of the cognitive unconscious and tracing their work back to newest insights in, for example, cognitive linguistics (cf., for an overview, Coëgnarts/Kravanja 2015b). Moreover, theories and approaches concerning embodied cognition continue to extend such studies in the direction of visual techniques, music and sound aspects, as well as of metaphors capable of stimulating the sensory-motor-affective level of experiences and enacting the construction of meaning structures (cf. Johnson in Coëgnarts/Kravanja 2015a: 12). In all cases, such mechanisms’ neurobiological underpinnings are a core concern (cf., e.g., Hasson et al. 2008; Jääskeläinen et al. 2008; Magliano/Zacks 2011; Gallese/Guerra 2012; and many more). One aim of these approaches that is in many respects analogous to the accounts presented in this book is then to “differentiate between the tools of the cinematic medium, as it is through the interaction of these tools that meaning is construed and the role of the embodied meaning-making resources becomes visible” (Coëgnarts/Kravanja 2015b: 36). This is very similar to the recognition of the integrative character of the filmic text we have argued for. However, we also take this further and consider the interaction of these ‘tools’ within the filmic text as establishing a common,

Conclusion

195

overarching framework that is capable of stretching across more traditional styles of film analysis, more recent approaches to the psychological study of film, and embodied cognition as well. All of these perspectives can be related to the integrative level of description and organization constituted essentially of textual patterns and guiding comprehension and interpretation. Such a construct may then not only serve to integrate traditional accounts of film semiotics and text analysis, but also to re-establish contact with other well developed views of complex multimedia artifacts and their socio-cultural as well as cognitive interpretation. A reorientation of this kind raises new possibilities for engaging again with some of the standard, but most illusive, topics of film theory and film interpretation – such as aesthetic and design choices, the use of intermedial and intertextual references or adaptation techniques – and for placing them on a firmer empirical basis by means of extensive corpus-based examinations and experimental studies. The combination of fine-grained material analyses, as shown in the first three chapters of the book, with higher-level interpretations, such as those exemplified in the latter chapters, prepares the ground for more reliable explorations of the whole process of film interpretation “from the perception to the comprehension of the film”, as required by an ever growing number of contemporary approaches working in the context of film studies today (e.g., Loschky et al. 2015: 1).

4 Conclusion – and an Invitation to Film Text Analysis We have suggested that our current understandings of text and textuality raise the bar concerning the possibilities offered for the close analysis of film. Such a claim is, of course, by no means self-evident and so it has been the goal of this book to support this argument by showing textuality and text at work during concrete film analyses. Each contribution to the book has considered facets of this general task, addressing a different and quite specific aspect of filmic textuality. The accounts have, on the one hand, returned to close analytic engagements with film and, on the other, also built on more contemporary developments concerning the nature of ‘texts’ as multimodal ensembles of expressive resources. Taken collectively the analyses illustrate how such a more theoretically focused but modally broader use of the notion of ‘text’ is beneficial for exploring how films function. As we have seen in the historical overview in the introduction to this volume as well as in the preceding chapters, an understanding of film analysis of this general kind has persisted in film studies over time and is certain to endure in the future. However, in order to provide a stronger foundation for the exploration of filmic meaning and its understanding – just as now also called for by Richard Dyer (2016), as we noted in our introduction to the volume – we have argued that it will be essential to go further and work with more recent developments concerning the dynamics of text and film. Only then, we believe, can the textual approach fulfill its promise, moving

196

Janina Wildfeuer & John A. Bateman

beyond loose or more metaphorical notions of film language, film grammar, film syntax, and so on. Finally, then, we consider the approaches presented here to show new potential for undertaking film text analysis, due to both their reliance on more contemporary understandings of text and textuality and to their adoption of the more refined analytic tools now available for supporting film research. Not only the example analyses, but also, similarly, the diverse contacts between disciplines, research areas, and interests establish many possibilities for future work. The approaches presented call particularly for theoretical evaluation and refinement as well as for further developments and applications within larger, empirical projects. Therefore, we invite all readers interested in the topic of film analysis to enter this discussion, to bring in their perspectives on film as text, and to link with the aspects introduced in this book by adopting and further developing the manifold ways of approaching film text analysis using textuality as an empowering mediator.

Bibliography Anderson, J. D. (1996). The Reality of Illusion: An Ecological Approach to Cognitive Film Theory. Carbondale and Edwardsville: Southern Illinois University Press. Bateman, J. A. (2013). “Dynamische Diskurssemantik als allgemeines Modell der Semiose. Überlegungen am Beispiel des Films”. Zeitschrift für Semiotik 35(3–4), pp. 249–284. Bateman, J.A./Schmidt, K.-H. (2012). Multimodal Film Analysis. How Films Mean. London/New York: Routledge. Bordwell, D. (1985). Narration in the Fiction Film. Madison: University of Wisconsin Press. Bordwell, D. (1989): Making Meaning. Inference and Rhetoric in the Interpretation of Cinema. Cambridge: Harvard University Press. Bordwell, D. (2004). “Neo-Structuralist Narratology and the Functions of Filmic Storytelling”. In: Ryan, M.-L. (ed.). Narrative Across Media. The Languages of Storytelling. Lincoln: University of Nebraska Press, pp. 203–219. Bordwell, D. (2008). Poetics of Cinema. London/New York: Routledge. Bordwell, D. (2011). “Common Sense + Film Theory = Common-Sense Film Theory?”. David Bordwell’s Website on Cinema: Essays. May 2011. Online: http:// www.davidbordwell.net/essays/commonsense.php [last accessed: 1 March 2016]. Coëgnarts, M./Kravanja, P. (eds.) (2015a). Embodied Cognition and Cinema. Leuven: Leuven University Press. Coëgnarts, M./Kravanja, P. (2015b). “Film as an Exemplar of Bodily MeaningMaking”. In: Coëgnarts, M./Kravanja, P. (eds.). Embodied Cognition and Cinema. Leuven: Leuven University Press, pp. 17–40. Dyer, R. (2016). “The Persistence of Textual Analysis”. Talk given as part of the Kracauer Lectures in Film and Media Theory at Goethe-University Frankfurt, Germany, 26 January 2016. Video Online: http://www.kracauer-lectures.de/de/ winter-2015-2016/richard-dyer/ [last accessed: 01 March 2016]. Gallese, V./Guerra, M. (2012). “Embodying Movies: Embodied Simulation and Film Studies”. Cinema: Journal of Philosophy and the Moving Image 3, pp. 183–210.

Conclusion  197 Grodal, T. (2009). Embodied Visions: Evolution, Emotions, Culture, and Film. Oxford: Oxford University Press. Hasson, U./Landesman, O./Knappmeyer, B./Vallines, I./Rubin, N./Heeger, D. J. (2008). “Neurocinematics: The Neuroscience of Film”. Projections: the Journal for Movies and Mind 2(1), pp. 1–26. Jääskeläinen, I-/Koskentalo, K./Balk, M./Autti, T./Kauramäki, J./Pomren, C./Sams, M. (2008). “Inter-Subject Synchronization of Prefrontal Cortex Hemodynamic Activity during Natural Viewing”. The Open Neuroimaging Journal 2, pp. 14–19. Kluss, T./Bateman, J./Preußer, H.-P./Schill, K. (2016). “Exploring the Role of Narrative Contextualization in Film Interpretation: Issues and Challenges for Eye-Tracking Methodology”. In: Reinhard, C.D./Olson, C.J. (eds.) Making Sense of Cinema: Empirical Studies into Film Spectators and Spectatorship. New York/ London: Bloomsbury Academic, pp. 257–284. Loschky, L./Larson, A.M./Magliano, J.P./Smith, T.J. (2015). “What Would Jaws Do? The Tyranny of Film and the Relationship between Gaze and Higher-Level Narrative Film Comprehension”. PLoS One 10(11): e0142474. DOI: 10.1371/ journal.pone.0142474. Magliano, J. P./Zacks, J. M. (2011). “The Impact of Continuity Editing in Narrative Film on Event Segmentation”. Cognitive Science 35, pp. 1489–1517. Mulvey, L. (1975). “Visual Pleasure and Narrative Cinema”. Screen 16(3), pp. 6–18. Reinhard, C.R./Olson, C.J. (eds.) (2016). Making Sense of Cinema. Empirical Studies into Film Spectators and Spectatorship. New York/London: Bloomsbury. Plantinga, C. (2009). Moving Viewers: American Film and the Spectator’s Experience. Los Angeles: University of California Press. Shimamura, A.P. (2013). Psychocinematics: Exploring Cognition at the Movies. New York: Oxford University Press. Staiger, J. (1992). Interpreting Films: Studies in the Historical Reception of American Cinema. Princeton: Princeton University Press. Tseng, C. (2013). Cohesion in Film. Tracking Film Elements. Basingstoke: Palgrave Macmillan. Wildfeuer, J. (2014). Film Discourse Interpretation. Towards a New Paradigm of Multimodal Film Analysis. London/New York: Routledge.

Filmography Eat, Pray, Love (2010). Ryan Murphy. Columbia Pictures Industries. Italy/India/ Indonesia/USA. Run Lola Run (Lola rennt, 1998). Tom Tykwer. X-Filme Creative Pool, Westdeutscher Rundfunk. Germany. Sliding Doors (1998). Peter Howitt. Intermedia Films, Mirage Enterprises. UK/ USA.

This page intentionally left blank

List of Contributors

John A. Bateman is Full Professor of Applied Linguistics in the English and Linguistics Departments of Bremen University, Germany, specializing in functional, computational, and multimodal linguistics. His research interests include functional linguistic approaches to multilingual and multimodal document design, dialogue systems, and discourse structure. He has been investigating the relation between language and social context for many years, focusing particularly on accounts of register, genre, functional variation, lexicogrammatical description and theory, multilingual and multimodal linguistic description, and computational instantiations of linguistic theory. He has published widely in all these areas, as well as authoring several introductory and survey articles on natural language generation, systemic-functional linguistics, and multimodal film analysis. His current interests center on the application of functional linguistic and corpus methods to multimodal meaning-making, analyzing and critiquing multimodal documents of all kinds, the development of linguistically-motivated ontologies, and the construction of computational dialogue systems for robot-human communication. Morten Boeriis is Associate Professor in Visual Communication at the University of Southern Denmark, Department of Language and Communication and is part of the Center for Multimodal Communication at SDU. He teaches a variety of courses on visual analysis at Business Communication studies and Film and Media studies at University of Southern Denmark. Other fields of expertise besides multimodality include media studies, systemic-functional linguistics, and media production. Morten Boeriis has for several years been specializing in multimodality both focusing on theory, description, and analysis. He has especially been interested in a multimodal approach to stills and moving images and in particular the grammar of e.g. photography, film, animation, social media, and sound design, etc. He has also been working in TV-production and as a freelance photographer. Neil Cohn is an American cognitive scientist internationally recognized for his research on the overlap of the structure and cognition of sequential images and language. He is Assistant Professor at Tilburg University,

200

List of Contributors

and received his doctorate in Psychology from Tufts University. Having originally started working in the comic industry as a teenager, his creative work includes several publications. In his academic work, Cohn introduces a broad framework for studying visual narratives in the linguistic and cognitive sciences. By bringing together scholars from across the globe in a recent edited volume, he aims at integrating interdisciplinary research into a unified field of studying visual narratives within the cognitive sciences. Joyce Goggin is a Senior Lecturer [UHD] at the Universiteit van Amsterdam where she teaches literature, and conducts research on film and new media. Proceeding from an economic perspective, her work addresses a variety of topics and media, such as gambling and speculation in film, painting literature, comics and video games, and she has published widely on related topics such as serialization, feminism, narrative, and convergence culture. While much of her work focuses on contemporary issues, Goggin has also published extensively on topics in the 17th, 18 th, and 19th centuries. Her most recent projects include an English translation and annotated edition of two Dutch plays about the South Sea Bubble written by Pieter Langendyk in 1720, and a collective project on the aesthetics and affects of cuteness in contemporary culture. Martine Huvenne is Lecturer and Researcher on Sound and Music for Film at the KASK-Conservatory, School of Arts, Gent. Her research and teaching are focused on the auditory part of the creative process of film-making. One of the main topics of her further research is the way sound in film guides the audience through film (in correlation with listening, filmic sound situates and choreographs the audience in filmic space) and an enactive approach to filmic listening. During the years following a Laban training course with Rob Stuyf (who himself was trained by Kurt Jooss), she organized dance workshops in which the interaction between music composition and movement composition was explored. Huvenne is curator and co-organizer of the Film Fest Gent annual Seminar on Music and Sound in Film. She coordinated the curriculum development of a European Master in Sound. Theo van Leeuwen is Professor at the Centre for Multimodal Communication as well as the Department of Languages (SG) and Communication, University of Southern Denmark, Odense, Denmark, and Emeritus Professor of Media and Communication at the University of Technology, Sydney, Australia. He is a social semiotician widely recognized as a co-founder, alongside Gunther Kress, of multimodality as an area of research in which he has published widely. His work has extended the influence of multimodality, social semiotics, and critical discourse analysis beyond semiotics, communication studies, and applied linguistics, to fields such as education, the arts, and media, culture and business studies.

List of Contributors

201

He is also known as a jazz pianist and film/TV editor, script-writer, and producer in the Netherlands and Australia. Janina Wildfeuer is a Researcher at Bremen University, Germany, in the areas of multimodal film and (digital) media studies as well as linguistics, discourse analysis, and semiotics. She teaches classes in multimodal, interdisciplinary, and applied linguistics and analyses film, comics, and other multimodal documents within several projects exploring the notion of multimodal discourse. Her publications include a monograph on film discourse interpretation, an edited volume building bridges for multimodal research as well as several contributions and papers on the analysis of multimodal artifacts, mostly focusing on interdisciplinary approaches and bridge building aspects in the humanities and beyond. Wolfgang Wildgen is Emeritus Professor of German and General Linguistics at Bremen University, Germany, where he worked as a full professor for more than 15 years on topics such as semiotics and visual semiotics, dynamics of communication, as well as semantics and sociolinguistics. His research currently focuses on questions of dynamic sign, language, and media theories, which he discusses as author and co-author of more than 20 books and a large number of research articles and book chapters in different languages.

This page intentionally left blank

Film Index

39 Steps, The (1935) 81 Adventures of Tintin, The. The Secret of the Unicorn (2011) 90 Alien (1979) 142 Aliens (1986) 142 Alter Bahnhof Video Walk (2012) 54

Forrest Gump (1994) 161 Frost/Nixon (2008) 160–2 Goldfinger (1964) 74 Gone with the Wind (1940) 26 Gravity (2013) 18–19, 46–8, 50–1, 56–7, 61–2

Battle of Los Angeles (2011) 156 Battleship Potemkin (1925) 78, 152 Blade Runner (1982) 142 Blair Witch Project, The (1999) 156 Bonny and Clyde (1967) 77–8 Breathless (1960) 79

Iron Man 2 (2010) 156–7 Ivan the Terrible (1947–1958) 25

Caligari (1920) 26 Casino Royale (2006) 18, 74 Close Encounters of the Third Kind (1977) 142 Cloverfield (2008) 156 Clueless (1995) 171 Cowboys & Aliens (2011) 142 Cries and Whispers (1972) 36 CSI Miami (2002–2012) 29–30

Magician, The (1958) 42 Man with a Movie Camera (1929) 53 Marathon Man, The (1976) 79 Mash (1970) 147, 149–50, 153 Matrix, The (1999) 19, 110–111, 152–4 Metropolis (1927) 73 Moonraker (1979) 74

Die Niebelungen: Siegfried (1924) 80 Dogville (2003) 72–3, 90 Dr. No (1962) 85 Duel (1971) 82

North by Northwest (1959) 68, 71–2, 81, 87

Eat, Pray, Love (2010) 169–183 Edge of Tomorrow (2014) 156–8, 160 El Sur (1983) 25 F for Fake (1975) 68, 90 Family Plot (1976) 68 Foreign Correspondent (1940) 81

Kiss me Deadly (1955) 37 La Course des Sergents de Ville (1907) 82

Ocean’s Eleven (2011) 173 Ocean’s Thirteen (2007) 173 Ocean’s Twelve (2004) 173 On Her Majesty’s Secret Service (1964) 76 Paranoid Park (2005) 51 Persona (1966) 36–7 Player, The (1982) 141 Pride and Prejudice and Zombies (2016) 142

204  Film Index Quantum of Solace (2008) 18, 80–6 Queen Christina (1933) 76–7 Rashomon (1950) 25 Rocky (1976) 79 Romeo & Juliet (1996) 171 Rope (1948) 67 Run Lola Run (1998) 189 Saddest Music in the World, The (2003) 158–9 Seventh Seal, The (1957) 35, 42 Shanghai Express (1932) 26 Shrek (2001) 19, 151–4 Simpsons, The (1989– ) 148 Skyfall (2012) 18, 81–2, 85

Sliding Doors (1998) 189 Spectre (2015) 76, 85 Spy Who Loved Me, The (1977) 76 Star Wars (1977) 18, 101–3, 105–6 Starship Troopers (1997) 154, 156 Suspicion (1941) 28, 31 Truman Show, The (1998) 72–3 Vampyr (1932) 35 Vanity Fair (2004) 171 Vertigo (1958) 71–2 Viridiana (1961) 148 Young and Innocent (1937) 89 Zelig (1983) 161

Name Index

Altman, Rick 51, 52, 62, 141 Andrew, Dudley 171, 172 Aristotle 105, 169 Arnheim, Rudolph 25, 26 Barthes, Roland 5–8, 12, 13, 122 Bateman, John A. 8, 15, 95, 99–101, 104, 108, 119–20, 128–30, 141–62, 191 Bazin, André 5, 123, 171 Bellour, Raymond 7–10, 122, 125 Bergman, Ingmar 35–7, 42 Bordwell, David 1, 3, 5, 8–15, 24, 50–2, 101, 105–6, 120, 127, 142–3, 152, 187–89, 191, 193 Buckland, Warren 96, 103, 118–20 Coëgnarts, Maarten 147, 187, 194 Cohn, Neil 18, 68, 94–112 Deleuze, Gilles 49, 53 Dyer, Richard 1–4, 15, 135, 195 Eco, Umberto 5 Eisentein, Sergei 4, 8, 11, 17, 25, 46, 53, 78, 94, 123, 152 Elsaesser, Thomas 30, 31, 40, 118 Fauconnier, Gilles 143–5, 147, 160 Forceville, Charles 147, 163 Gilbert, Elizabeth 173–178 Greimas, Algirdas 8, 88 Halliday, Michael A.K. 12, 17, 32, 41, 122, 124–6 Heath, Stephen 9, 10 Hitchcock, Alfred 18, 28, 66–8, 72, 81, 87–9 Hjelmslev, Louis 7–9, 12, 88 Jackendoff, Ray 96, 97, 100 Jakobson, Roman 7, 70

Kappelhoff, Hermann 82, 147, 150, 153 Kracauer, Siegfried 26, 82 Kravanja, Peter 147, 187, 194 Kress, Gunther 17, 27, 32, 33, 36–40, 122–5, 158 Lakoff, George 145, 160 Lévi-Strauss, Claude 27 Manovich, Lev 30, 53–4 Merleau-Ponty, Maurice 18, 49, 52, 56, 62 Metz, Christian 5, 7–11, 70, 96, 100–1, 105–6, 118, 122 Murnau, Friedrich Wilhelm 26 Pasolini, Paolo 5, 9 Peirce, Charles Sanders 120 Pudovkin, Vsevolod 4, 5, 8, 11 Saussure, Ferdinand de 8–14, 119, 187 Sloterdijk, Peter 52, 57, 62 Sobchack, Vivian 47, 50 Staiger, Janet 141, 143, 194 Stam, Robert 171 Thompson, Kristin 8, 24, 25, 50–2, 62, 101, 105, 141 Turner, William 143, 145, 147, 160 van Leeuwen, Theo 17, 24–44, 99, 123, 125, 130, 158 Vertov, Dziga 4, 53 Wark Griffith, David 4 Wollen, Peter 15 Wuss, Peter 86 Zacks, Jeffrey M. 94–96, 104–5, 107, 111, 194

This page intentionally left blank

Subject Index

action 16, 18, 25, 34–6, 66–9, 70–5, 88, 97, 152–5; bodily action 67; social action 67; parallel action 80, 85 adaptation 19–20, 88, 169–179; expanding adaptation 174; expanded adaptation 19, 174, 180; film adaptation 108, 170–183 aesthetics 3, 74, 169–70, 173, 193 blending 19, 141–64 bricolage 27, 31, 141 closure 67, 94 code 9–11, 30, 67, 71, 87, 119, 173, 187; see also semiotic code cognition 5, 14, 17, 70, 89, 111, 160, 195 cognitive process 68, 76, 94–6, 104, 160, 187 common sense 1, 187–9, 192 comprehension 94, 106, 108, 127; film(ic) comprehension 18, 94, 112; image comprehension 95; narrative comprehension 18, 94, 112, 127, 188, 193–5 context 12, 19, 25, 33–4, 70, 75, 77, 97, 118–27, 130–5, 173, 187 discourse 11, 14–15, 18, 39, 67, 94, 96, 120, 122–7, 150, 153, 178, 189–93; filmic discourse 14, 19, 121, 125–6, 132 discourse relations 130, 132, discourse semantics 129–30, 132–5 dynamic 12–16, 47–9, 66, 83, 87, 122–4, 131–133, 152–4, 178; physical dynamic 69–76 economy 67, 173, 177, 182 event 18, 40, 53–5, 66, 73–4, 78, 86, 104, 109, 111, 153, 161, 188, 191

event segmentation 94 fabula 38, 188, 193, film noir 24, 35, 40 film semiotics 11, 120, 121, 190, 195 film text analysis 1–4, 12–16, 189–190, 195–6 film theory 4, 13, 24, 120, 122–3, 187, 190, 192–3; cognitive film theory 14, 135, 192 filmic text 1, 4, 9–12, 14, 18–20, 118, 120–7, 135, 190–5 genre 32, 82, 141, 142–4, 155–6, 162, 178, 182 grammar 8, 18, 25, 30, 31–40, 96–7, 106, 112, 193; filmic (narrative) grammar 94, 106–7, 196; grammar of lighting 30, 32–40; visual (narrative) grammar 94, 96, 107 grande syntagmatique 8, 10, 100 images 30, 33, 36–40, 50, 53, 88, 94–6, 107–110, 130, 157; moving images 55, 94, 106–7, 112; sequential images 18, 94–6 inferences 15, 42, 95, 97, 101, 120, 127, 130–1, 145, 147–9, 188; abductive inferences 19, 121, 135 inferential process 15, 17, 94, 120, 194 intersemiosis 124, 127–9 kinesphere 57, 59–63 language 4–5, 24, 28, 41, 53, 66, 70–2, 80, 96, 100, 108, 122, 171; film language 4, 196 lighting 17, 24–45, 79, 150, 191; computer-rendered lighting 30, 32; narrative lighting 25, 31; textbook lighting 28, 32, 42

208

Subject Index

linguistics 6, 9–12, 14–15, 118, 122, 193–4 literary approaches 2, 5, 7, 14, 74, 135, 170, 190–3 material 1, 3, 8, 13, 15, 17–9, 27, 34, 42, 68, 118, 120–5, 128, 135, 157–61, 191, 194; material manifestations 2, 13, 16 meaning 1, 4, 16–20, 24–5, 33–5, 39–43, 73, 80, 96, 104, 118–33, 141, 145–6, 191–6 meaning making 16, 19–20, 27, 34, 119–23, 162, 187, 193 meaning potential 17, 25, 34–5, 39–42, 196 mediality 16; intermediality 16, 141, 143–5, 149–53, 162, 191; transmediality 162 metafunction 17, 32–6, 43 metaphor 35, 88, 142, 144–7, 153, 178, 196; Conceptual Metaphor Theory 145, 159 montage 4, 5, 11, 46, 53–4, 67–8, 79–80, 152–3 multimodal 2, 15, 17, 36, 43, 107, 153, 194 multimodal (discourse) analysis 15, 120, 122–4, 130, 134, 190 multimodal text 43, 121–2, 124–5 narrative 8, 25, 52–3, 70, 127, 130, 132, 150, 153, 163, 170, 174–6, 180, 187–9, 193; visual narrative 18, 94–7, 104–9 narrative schema 97–8, 100 narrative structure see structure perception 17, 27, 39, 47–50, 53–6, 75, 94–5, 107–9, 111–112, 127, 170, 181, 187–8, 194–5; audio-visual perception 18, 46, 54–55, 187; embodied perception 15, 194 phenomenology 18, 46, 49–50, 54, 75, 194 physics 85, 88, 176; movie physics 18, 66, 68–70, 75–6, 81; physics of action 74; semiophysics 18, 69 pragmatic 19–20, 118–21, 127–9, 133–5, 163, 192 realism 5, 27, 69, 74–5, 156, 178; content realism 75; physical realism 74–5; process realism 75

reasoning 120, 145–7 recipient 14–17, 118–21, 127, 128–33, 160, 187; see also spectator semantic 36, 94, 95–7, 99–102, 119–21, 127, 129, 131–36, 146, 189, 192; see also discourse semantics semiology 118 semiotic code 10 semiotic mode 17, 25, 32–3, 43, 124, 130, 191 semiotic resource 16–17, 25, 34, 119–20, 124–6, 130, 143 semiotic system 5, 70, 88, 171 signs 10, 13–14, 16, 27, 33, 70, 88, 124, 173; iconic signs 15 social semiotics 17, 25, 32 sound 33, 46–62, 66, 108, 128–9, 131, 158, 161, 191 sound concept 46–8, 62 space 8, 16, 37, 46–62, 70, 71–5, 81–2, 87, 109, 128, 131, 145–50, 178, 192; dynamic space 46–8; editing space 46, 53–7; filmic space 46, 48, 52, 57; input space 145–50, 159–60; internal diegetic space 62; lived space 46, 48, 50–2, 55–7; musical space 50; smooth space 49, 51 spectator 1, 54 see also recipient structure 70, 73, 77, 80, 94, 100, 103–7, 120, 123–5, 127, 129–30, 144–9, 158–60, 178, 191; narrative structure 8, 18, 20, 80, 94–5, 109, 111 sublime 133–4 text 1–20, 33, 42–3, 67, 80, 118–35, 156, 158, 163, 169, 170–3, 187–96; notion of text 2, 10, 15–6, 120–1, 190–4; see also multimodal text, filmic text text(ual) analysis 1–3, 16–7, 119, 134 textual logic 8, 187 textual patterns 10, 192–3, 195 textual qualities 2, 129, 134–5, 191, 193 textuality 2, 3, 7, 12, 15, 121, 128, 163, 180, 188–93, 195–6 texture 16, 158, 192 understanding 3–7, 12, 20, 39, 68, 75, 95, 122–4, 146, 187–90, 196