Digital Synesthesia: A Model for the Aesthetics of Digital Art 9783110459937, 9783110459340

Table of contents :
Contents
Foreword
Introduction
Part 1. Theories on Digital Synesthesia
Syn-Aesthetics of Digital Art
Synesthesia: Perception, Language, Digital Art
A Philosophical Perspective on Unified Conscious Experience in Synesthesia: Insights from Philosophy of Perception and Aesthetics
Ideasthesia and Art
Transformation, Transliteration and Translation. Synesthesia and Multisensory Perception in Contemporary Visual Art
Sensing Digital Art: Aesthetic Acts Beyond (the two) Synesthesia(s)
Digital Synesthesia: Reminiscing about Science and Art
The Incidental Synesthete
The Interaction between Seeing and Moving: Eye Movements as a Window to the Mind?
Digital Synesthesia
Part 2. Artistic Research on Digital Synesthesia
VERTICAL 2
Space Time
At Play
Diver
E.E.G. KISS
The Flying Umbrella Project
Bestiary for the Minds of the 21st Century: Genomic Opera
Sound Calligraphy
facades
MotU #4–#6
Topography of Movement
IN_SIDE VIEW
Transmission+Interference
I am Sound
Data Music
Part 3. Digital Synesthesia Exhibition
Angewandte Innovation Laboratory, Vienna, 2016
Authors' Biographies
Imprint

Citation preview

Digital Synesthesia

Edition Angewandte Book Series of the University of Applied Arts Vienna Edited by Gerald Bast, Rector

Digital Synesthesia

Katharina Gsöllpointner Ruth Schnell Romana K. Schuler (Eds.)

A Model for the Aesthetics of Digital Art

Contents 06 – 07  Foreword 

Gerald Bast 08 – 09 Introduction 

Katharina Gsöllpointner, Ruth Schnell, Romana K. Schuler

Part 1

11 – 28   Katharina Gsöllpointner

Theories on Digital Synesthesia

Syn-Aesthetics of Digital Art 29 – 34  L awrence E. Marks / Catherine M. Mulvenna

Synesthesia: Perception, Language, Digital Art 35 – 40  Aleksandra Mroczko-Wąsowicz

A Philosophical Perspective on Unified Conscious Experience in Synesthesia: Insights from Philosophy of Perception and Aesthetics 41 – 52   D anko Nikolić

Ideasthesia and Art 53 – 58   Regine Rapp

Transformation, Transliteration and Translation. Synesthesia and Multisensory Perception in Contemporary Visual Art 59 – 64  C hris Salter

Sensing Digital Art: Aesthetic Acts Beyond (the two) Synesthesia(s) 65 – 80 Romana K. Schuler

Digital Synesthesia: Reminiscing about Science and Art 81 – 84   J effrey Shaw/Sarah Kenderdine

The Incidental Synesthete 85 – 90   M iriam Spering

The Interaction between Seeing and Moving: Eye Movements as a Window to the Mind? 91 – 98  Peter Weibel

Digital Synesthesia 

Part 2

Artistic Research on Digital Synesthesia

102 – 105 Anke Eckardt

VERTICAL 2 106 – 109 Karl Heinz Jeron

Space Time 110 – 113   kondition pluriel

At Play 114 – 117   kondition pluriel

Diver 118 – 123   Karen Lancel / Hermen Maat

E.E.G. KISS 124 – 125   A lan Kwan

The Flying Umbrella Project 126 – 131  Marcello Mercado

Bestiary for the Minds of the 21 st Century: Genomic Opera 132 – 135   U lla Rauter

Sound Calligraphy 136 – 139   R uth Schnell

facades 140 – 143   R uth Schnell

MotU #4–#6 144 – 147   R uth Schnell

Topography of Movement 148 – 149   J effrey Shaw/Sarah Kenderdine

IN_SIDE VIEW 150 – 153  David Strang / Vincent Van Uffelen

Transmission+Interference 154 – 157   Tamiko Thiel / Christoph Reiserer

I am Sound 158 – 162  Peter Weibel

Data Music

Part 3

Digital Synesthesia Exhibition

164 – 183  Angewandte Innovation Laboratory,

Vienna, 2016

186 – 197  Authors’ Biographies 198 – 199  Imprint

Foreword

6

Gerald Bast

In his essay “The Future of The Image” 1 Jacques Ranciére poses the question, “What is being spoken about and what precisely are we being told, when it is said that there is no longer reality but only images? Or, conversely, that there are no more images but only a reality incessantly representing itself?” The images that we encounter everywhere today are mostly digital realities, projections of binary codes. The codes represent only themselves; reality is reduced to the combination of minimalisms. However, the resulting complexity of the world we experience appears to contradict this. Digitally created realities are often referred to as “soulless” and this criticism is based on a lack of complexity. Nonetheless, “The word is only made flesh through a narrative. An additional operation is always required to transform,” argues Ranciére. The same can be said about images. The image is the original form of the word, the spoken or written word referring to facts. “The logical picture of the facts is the thought,” proposes Wittgenstein in his tractatus logico philosophicus. When the picture of reality is however a result of a transformative operation, then the criticism of “soullessness” is not actually directed at the picture itself, but rather at the operation involved in creating the picture, the superposing of a narrative onto the resulting combination of binary codes. It is not the images that we produce of reality – or rather those that are imprinted on us – that are lacking in complexity, but the accompanying narratives. The one-dimensionality of thoughts, images and verbal arguments that impinges on us everywhere and which prompts us to despair at the complexity of (social and economic) facts is not owed to digitalization, but rather indicative of our insufficient ability to merge various aspects of perception. Consequently, generating complexity is a synesthetizising process. Awareness of the effects and opportunities of synesthetizising processes in a digital world characterized by both overwhelming complexity and one-dimensionality can best be raised – as is so often the case – through arts-based research.

1 Original: Le destin des images (2003)

This makes the contributions of the artists involved in the artistic research project DIGITAL SYNESTHESIA immeasurably valuable.

7

Introduction

The articles in this publication represent the artistic and scientific outcomes of the arts-based research project DIGITAL SYNESTHESIA, which was conducted from 2013 to 2016 at the Department of Digital Art at the University of Applied Arts Vienna. This project, which had the aim of researching the synesthetic capabilities of digital art, was the result of a collaboration of media artists, media and art scholars as well as neuroand cognitive scientists. The starting point of the inter- and transdisciplinary project was the observation of new individual and social modes of perception which appear to be linked to the culture of the digital world. Cultural, and therefore social and individual, modes of perception per se are always historically linked to new communications technologies and media. In the context of this work, the question is raised of how the multimedia aesthetics of digital technologies is changing human perception and how digital art can explore, represent and reflect on these changes. The thesis that digital art can — mainly due to the binary code that distinguishes it from all other arts — produce synesthetic perceptual experiences in the users marked the beginning of the project. On account of its binary code basis, the direct binary translation and transformation of various forms of one medium into another — such as sound into image or text into kinetics — is only possible with digital art. Over the course of the research process, however, the multimodal perception qualities of genuine multimedia-authored digital art have increasingly come to the foreground. The focus has shifted, so to speak, from the digital to the analog nature of digital art, in which ‘analog’ is understood as a supplementary term to ‘digital’ as opposed to a contrasting one. The ‘analog’ characteristics of digital art are above all evident in the simultaneity and entwinement of visual, auditory, olfactory, gustatory, haptic, tactile and kinesthetic media, making syn-aesthesia (a combination of several levels and areas of perception) possible at the perception level. This description of the aesthetics of digital art, focusing now far more on the nature of algorithms, or on the computer program, showed a consensus with current neuro- and cognitive scientific research on synesthesia, which in the sciences as well is no longer defined as merely cross-modal, but also as an intermodal perceptual phenomenon that is even shaped by semantic and social parameters. The research processes of the 17 international media artists involved in DIGITAL SYNESTHESIA evidently present themselves as the artworks created by these artists in the course of the project. The 15 art projects in total are described in this book from the perspective of the artists in terms of their aesthetic and communicative concepts and in relation to the theme of DIGITAL SYNESTHESIA. To highlight the theme of the research area, “primary sensory modalities”, which are supposed to reflect the diversity of the synesthetic perceptual qualities of the projects, are also assigned to each artistic project.

8

A separate section also documents the exhibition of the artworks on display at the Angewandte Innovation Laboratory in Vienna in the spring of 2016. For an overview of the current situation in neuroscientific, art- and media-theoretical research on the topic of synesthesia and digital art, scientists and theorists were invited to present the state of the art by showcasing examples of their respective findings. These articles furthermore represent the proceedings from the DIGITAL SYNESTHESIA international symposium, which was held at the University of Applied Arts Vienna in April 2016. The special aesthetic of the book cover is a reference to the multimodality of synesthesia. The words that appear on the cover are a compilation taken from DIGITAL SYNESTHESIA artwork titles and from the keywords of the theoretical articles. The terms on the inside of the cover are a collection of the senses, sensory modalities and sensory domains, and of the classifications of currently known types of synesthesia. The multimodal and multimedia aesthetic of the cover thus also represents the sensory interpretation of the findings of the artistic research project DIGITAL SYNESTHESIA. Acknowledgements We want to thank the artists of the DIGITAL SYNESTHESIA project for the production of their exciting artworks as research projects, and the scientists and scholars for the contribution of their valuable research texts. We’d also like to thank the curators who, with their media art expertise, supported the international DIGITAL SYNESTHESIA Call in 2014: Inke Arns, Charlotte Bank, Lee Ann Biddle, Josephine Bosma, Christa Blümlinger, Timothy Druckrey, Severin Dünser, Kathy Rae Huffman, Erkki Huhtamo, Geert Lovink, Ludwig Seyfarth, Mitchell Whitelaw. We are especially grateful to Jeffrey Shaw and Peter Weibel for their highly valuable support as DIGITAL SYNESTHESIA cooperation partners and as members of the DIGITAL SYNESTHESIA curatorial board. Both their extraordinary expertise in the field of media art and their invaluable advice for the selection of the participating artists has been of great importance to the success of the project.

Katharina Gsöllpointner Ruth Schnell Romana K. Schuler

9

THEORIES ON DIGITAL SYNESTHESIA Part 1

This book chapter combines ten theor which have been written by neuro- an media and art theorists. The articles re between media aesthetics, synesthetic modalities, theories of embodiment, d issues in the context of DIGITAL SYNE of the international DIGITAL SYNESTH University of Applied Arts Vienna on A

10

Syn-Aesthetics of Digital Art

Katharina Gsöllpointner

 Keywords   embodied perception, synesthesia, media aesthetics, digital art, kinesthesia, cross-modality, sensory domains, ideasthesia, language, metaphor

retical texts on DIGITAL SYNESTHESIA nd cognitive scientists as well as by efer to topics regarding the relationship c perception, brain activities, sensory digital technologies, and philosophical ESTHESIA. They are also the proceedings HESIA symposium which was held at the April 4 and 5, 2016. To arrive at a complete picture of synesthetic mental Image-schemas are general topological and oriental processes it is important to be aware that synesthetic

structures that are kinaesthetic in nature. They have an

metaphors are the result of verbalized synesthetic ex-

analog rather than digital character.

periences, but that in synesthetic cognition, language

(George Lakoff 1987)

is not the only modus operandi. Although synesthetic poetry and synesthetic painting have been described

By the very change of visual sensation, which we

in detail, attention to synesthetic concepts in the form

register when we move the locality of our body, the

of mediated visual, auditory or audiovisual imagery

notion of depth of space emerges. If we are unable to

has been rare. All new technologies — light, film,

move, there would be no motor system. If we were not

electronic sounds, images, algorithm — have inspired

able to control the movement of our motor system, we

artists to compose new relations between the arts and

would not be able to perceive the sensation, or con 12

to evoke new synesthetic experiences in an audience.

Optatibeatespace. lamet optiis dolor quossequi dolum struct a three-dimensional We allaudi experience

(Frans Evers 2012)

this phenomenon. Poincareé presented us with a little

quos dolupta ecaeria

philosophical curlyque and said, “My friends you need motion in order to perceive.” (Heinz von Foerster 1977)

11

Digital Art and Synesthetic Perception Synesthesia is a striking example of the brain’s ability

synesthetic perception of digital artworks? How do

to cross-modally integrate two or more sensory

media influence the sensations of the recipients and

modalities, dealing with phenomena such as sounds

users? How are the sensory modalities of one medium

being perceived as strong visual images, or even

(e.g., light and dark in image, silent and loud in sound,

objects like chairs being tasted. For a long time

bitter and sweet in taste, fresh and fetid in smell, soft

synesthesia was considered to be an “anormal” mani-

and hard in haptics/tactiles) transferred to another?

festation of perception, but during recent decades

Is there something other than the five classic sensory

it has seemed more and more apparent that

domains (vision, audio, taste, smell, touch) involved

synesthesia must be a phenomenon which is only

in the construction of synesthetic perception? Can

a concentration and an intensification of an

synesthesia-inducing stimuli be detected by means

explore the capabilities of digital art to create translational and cross-modal sensory experiences

of the various media forms of an artwork? Are certain media features especially appropriate for evoking synesthetic experiences? On the side of the aesthetic appearance of art-

and to provide synesthetic experiences for non-

works it is the artistic media, their modalities, and

synesthetes. These presumptive capabilities are

their forms which provide information about the

based on the unique and special technological and

interaction and the intertwinedness of perception

media conditions of digital art, namely on its binary

properties which induce synesthetic experiences.

code which not only allows for the direct and immedi-

These media forms manifest themselves within various

ate translation of outputs in one sensory domain (e.g.,

parameters, which I will discuss further below. Here

pixel for visual representation) into outputs in another

it is important to stress the fact that the artistic act per

(e.g., sound units for auditory representation), but

se is to bring the media into forms, i.e., to create an

above all allows for the programming of algorithms.

artwork by means of what I call aesthetic decisions.

The programming represents the artistic and aesthetic

As I have shown in earlier research contexts, artists

basis for those qualities of digital art which differen-

have an expertise about aesthetics. They have a

tiate it from all other art (including analog electronic

know-how in regard to aesthetic decisions which,

media art, video and film, digital video and film

amongst other things, includes expectations about

and their postproduction). Only the code enables

effects and impacts on the perception of their work

man-machine-interaction, machine-machine-

(Gsöllpointner and Moser 2009). Permanently making

interaction, and the machine networks, all three

these aesthetic decisions about media and their

of which are pre-conditions for digital art’s strong

forms, artists therefore develop an expertise in terms

spatiotemporal constructedness. It was a main presumption of the project that

of aesthetics; they gain an Aesthetic Know-how which is not only theoretical but above all practical, based

inter- and transdisciplinarity between artists, scholars

on the experience of conceiving, producing and

and scientists must be the essential method for ex-

manufacturing artworks. The aesthetics of digital

ploring the fascinating field of synesthesia from the

artworks concede the recipient/user of the artwork

perspective of artistic research. The mutual exchange

a much higher degree of freedom to deal with than

of knowledge between art, humanities and sciences

do non-digital artworks. It is not for nothing that the

opened new perspectives within the respective fields.

terms “interactive” (e.g., in Peter Weibel’s installation

For media studies, and here especially for media

Data Music [Fig. 1] and in Tamiko Thiel & Christoph

aesthetics, the role of the media in the exploration

Reiserers’ work I am Sound [Fig. 2] ), “immersive” (e.g.,

of synesthesia is of central interest. How exactly do

in Jeffrey Shaw & Sarah Kenderdine’s IN_SIDE VIEW

media (in their respective forms) contribute to the

[Fig. 3] ) or “responsive” (e.g., in kondition pluriel’s At

THEORIES ON DIGITAL SYNESTHESIA

Katharina Gsöllpointner

The aim of DIGITAL SYNESTHESIA has been to

12

Syn-Aesthetics of Digital Art

otherwise “normal” perceptive processing.

Play [Fig. 4] ) are fundamental to the description of

cross-, multi- and trans-modal character in this

artistic media in artworks. Digital artworks can there-

model is on the one hand determined by the sensory

fore function as best practice examples of the process

constitution of perception, but also on the other by

of perception per se, not just because of their “beau-

neuroscientific findings about synesthetic processing

ty” or other psychological features, but because they

which have revealed a strong involvement of

offer the possibility of observing “pure” forms and the

conceptual parameters in perception as a holistic

media of the latter.

bodily process. Thus, the traditional differentiation

To explore the media functions in “syn-aesthetic”

between “strong” and “weak” synesthesia (Martino

digital art, I formed a model to compare the multi-

and Marks 2001, Marks and Mulvenna 2013) might

media features of digital artworks with the multi-

have become obsolete as research has revealed

modal features in types of synesthesia by describing

new knowledge about the involvement of what

their syn-aesthetic correlations (Gsöllpointner 2015).

scientists call “semantic inducers” in synesthetic

I have suggested that digital art and synesthesia

experiences. Semantic inducers are non-sensory

can form ideal correlatives to explore their media or

spatial, temporal, and motional concepts, such as for

modal constitutions respectively by means of

instance letters, numbers, calendar units, and even

comparing the features of an artwork ­– such as its

swimming styles. They do not count as sensory

title, its primary and secondary artistic media, its

inducers but can still evoke synesthetic experiences

cross-modal aesthetics, or its possible semantic

(cf. Amin et al. 2011, Hale et al. 2014, Mroczko-

­ with correlating features of synesthesia associations –

Wąsowicz and Nikolić 2014).

like the type’s name, its primary and secondary

In the context of the multimedia conditions

sensory domains, its cross-modal effects, or its

of digital art, these semantic inducers could also be

possible semantic inducers. Fig. 5 shows how

seen as representative of a “forgotten” domain of

modal components of synesthesia can be matched

the senses. Such a “sixth sense” (Berthoz 2000,

with media components of digital artworks in terms

Fingerhut 2011) not only extends the area of the

of aesthetic correlations and therefore give

traditional five human senses but could even be

information about how forms of media contribute

considered as encompassing and integrating all

to the understanding of perceptive phenomena.

physical perceptive and cognitive processes by its

The connectedness between digital media, their forms in artworks, synesthetic perception and its

literally embodied structure: the kinesthetic domain or kinesthesia.

Synesthesia and Kinesthetics Kinesthesia as a unique sensory domain has been

Webster Dictionary 2015). However, in science there

rediscovered by the sciences in recent years, although

still lacks a consistent and standardized definition

its existence has been under investigation since as

of the term.

early as the 19th century. The exact definition of kines-

Historically, the term kinesthesia derives from early

thesia (the kinesthetic sense) is still controversial and

studies about the physiological conditions of humans

therefore varies in literature. In general, the following

and animals moving their limbs and staying in balance.

very brief description provides a good conclusion

It has also been called the “sense of locomotion” (J. C.

of the basic points: one can say that kinesthesia is

Scaliger 1557) or the “muscular sense” (C. Bell 1826).

“a sense mediated by receptors located in muscles,

H. C. Bastian proposed the term kinesthesia in 1880,

tendons, and joints and stimulated by bodily move-

and in 1906 C. S. Sherrington introduced the terms

ments and tensions”; but it is also at the same time the

proprioception, interoception and exteroception as

“sensory experience derived from this sense.” (Merriam

specifications of the “locality” of the input of sensory

13

Syn-Aesthetics of Digital Art

Fig. 2

THEORIES ON DIGITAL SYNESTHESIA Katharina Gsöllpointner

Fig. 1

Fig. 4

14

Fig. 3

Structural Correlations between Synesthesia and Digital Art

Fig. 1 Peter Weibel: Data Music, 2016 (Exhibition view AIL Vienna) Photo: Birgit and Peter Kainz

Fig. 2 Tamiko Thiel / Christoph Reiserer: I am Sound, 2016 (Exhibition view AIL Vienna) Photo: Birgit and Peter Kainz

Fig. 3 Jeffrey Shaw / Sarah Kenderdine: IN_SIDE VIEW, 2016 (Exhibition view AIL Vienna) Photo: Birgit and Peter Kainz

Fig. 4 kondition pluriel: At Play / Enjeux (Performance at Theatre La Chapelle, Montreal. Photo: Bruno Colpron, © kondition pluriel, 2015)

Fig. 5 Syn-aesthetic correlations between synesthesia and digital art (Gsöllpointner 2015)

Synesthesia (Neuroaesthetic components)

Digital Art (Aesthetic components)

Type

Title

Synesthetic effect(s) (Phenomena)

Aesthetic effect(s) (Description)

Involved primary sensory domains (e.g., auditive, visual, kinesthetic, etc.)

Involved primary (artistic) media (e.g., sound, image, spatial installation, etc.)

Secondary sensory domains

Secondary (artistic) media

Main directionality from stimulus inducer to (concurrent) sensory evocation (Additionally bidirectionality has been proven also, e.g., by Ward et al. 2008, 1295)

Main directionality from one medium to another (Bi- and multidirectionality or simultaneity is often a matter of course; intermedia)

Multimodal aspects (Involvement of many / all sensory modalities)

Multimedia aspects (e.g., media embodiment, “Gesamtkunstwerk”, etc.)

Cross-modal effect(s) (A stimulus from one sensory domain is experienced in several sensory domains)

Cross-modal aesthetic(s) (Simultanous, computerized translation from one medium into one or many others, by means of binary code)

Intermodal quality correlations of inducers and concurrents (e.g., auditive qualities like pitch correlate with visual qualities like brightness)

Intermodal quality correlations of different media (e.g., metaphorical comparisons like “sound is light”, language metaphors, etc.)

Applicable exemplary semantic inducer(s) (e.g., personification of inducers, mirror touch inducers, etc.)

Possible semantic associations (Semantic field)

Involved spatiotemporal concepts (e.g., movement of concurrents, time units, numbers, acceleration and speed, etc.)

Spatiotemporal components (e.g., [moving] pictures, installations which require the movement of recipients, time-based artworks like film, video, interactive installations, net.art, music, game art, etc.)

Fig. 5

15

stimuli. Today it has been proved that on the physio-

system consists not only of immediate skin contact

logical level, muscular nervous activities are definitely

and tactility but also internal bodily, or somatic,

involved with the kinesthetic sense in manifold ways

sensations. Proprioception constitutes one of these

(Proske and Gandevia 2009).

somatic sensations within the haptic system. In addition to proprioception, these somatic sensations comprise kinesthesia (the sense of movement) and the

can be described, Fulkerson has suggested the

vestibular system (the sense of balance).” (Storks 2016)

model of “sensory pluralism” (Fulkerson 2014). Instead

In this model the kinesthetic sense would — together

of the dichotomy of a (conservative) concept of

with the vestibular sense (equilibrioception or the

“sensory monism” which represents a model deter-

sense of balance) — be part of the proprioceptive sense

mined by single, distinguished sensory domains on

which, as mentioned above, is connected strongly to

the one hand, and the complete elimination of the

the haptic or tactile sense. Moreover, kinesthesia also has a special, strong connection with the visual sense. Berthoz explains

that rejects any single, unified account of sensory

this interdependency of vision and kinesthetics in

modalities and their interactions, instead embracing a

reference to Merleau-Ponty’s expression “La vision est

multitude of such accounts.” (Fulkerson 2014). Already

palpation par le regard” (“Seeing is touching with the

Gibson has suggested that the senses are a “perceptual

gaze.” Translation into English by the author) (Berthoz

system” rather than a conglomeration of single senses,

2014). Spering and Carrasco, for instance, have shown

and he referred to kinesthetics as to “cut across the

how tiny eye movements (saccades) are a necessary

functional perceptual system” (Gibson 1966, cit. in

pre-condition for visual perception per se (Spering &

Fingerhut 2011). The concept of kinesthesia as an inte-

Carrasco 2015). And in the context of art perception,

grating sense would fit well into these considerations.

O’Regan and Noë also explored the sensorimotor

In general it is clear that, in addition to the classic five so-called exteroceptive senses – seeing (visual), hearing (auditory), tasting (gustatory), smelling

account of vision and visual consciousness early on (O’Regan and Noë 2001). Very often though, the term proprioception is used

(olfactory) and touching (tactile/haptic) – humans

instead of kinesthesia, and is described as the “sense

have a range of other sensory perception capabilities,

of position and posture, movement and velocity of

which are in general subsumed under the concept of

the body and body parts.” (Storks 2016) Storks con-

interoception, which again contains the so-called

cludes the manifold approaches to the exploration of

proprioception, equilibrioception, visceroception,

kinesthesia and the subsequent confusion, ambiguity

thermoception, and nociception. In this model, visceroception means the perception

and almost nebulosity of the meaning of the concept of kinesthesia in science as follows: “Paterson … con-

of internal organs like hearing the blood flow through

siders kinesthesia and proprioception to be equal

the veins, feeling one’s heartbeat or sensing one’s di-

components of somatic sensations and notes that they

gestion. Thermoception is the sensing of heat and cold,

work in conjunction with the vestibular system … Ham-

and nociception means sensing pain, both

ilton points out that proprioception comprises kines-

of which are sensed by the same receptors in the

thesia as well as the knowledge of fatigue and warmth

body (Proske and Gandevia 2009). Both of the latter

and cold, the vestibular system, the visceral sense

are also considered to be very closely attached to

and visual proprioception (the kinesthetic function of

the haptic sense of touch. However, this is also

vision) … Berthoz, in turn, classifies proprioception as

claimed for the term proprioception: “Proprioception …

a component of kinesthesia which also includes the

originally stems from the fragmentation of the sense

vestibular system and vision …“ (Storks 2016).

of touch. As such, proprioception is still classified as a component of touch or the haptic system. The haptic

THEORIES ON DIGITAL SYNESTHESIA

Katharina Gsöllpointner

concept of distinguishable senses as “eliminativism” on the other, he argues for “an intermediate view

Basically, I would like to follow Berthoz’ suggestion and subsume proprioception and the vestibular sense

16

Syn-Aesthetics of Digital Art

In the course of the ongoing academic discussions about the conception of senses and how exactly they

under the term of kinesthesia. Kinesthesia can thus

Such semantic inducers cannot be ascribed to

be simply described as “the sense of moving”

sensory stimuli origins like images, sounds, smells,

or “the moving sense.” Furthermore, this also fits in

tastes or haptics/tactiles. They are merely conceptual

with Wolve’s use of the term as distinctive to a

ideas or imaginations which induce cross-modal

concept of a dis-embodied mind: “Here, the word

perception and which are mainly temporal, spatial,

‘proprioception’ will serve as a short-hand

motional and social concepts. Semantically induced

designation for the priority of dynamic embodied

synesthesia includes, for instance, phenomena like

activity over isolated ‘mental’ and ‘physical’ regions.

mirror-touch synesthesia (Banissy and Ward 2007),

Rather than asking ‘How can a brain accomplish

or the personification of graphemes (Amin et al.

reasoning?’, the question becomes ‘How can a

2011). It has also been shown that the cognitive

brain have experiences?’, that is, ‘What is it like for

activities of the brain alone can evoke synesthetic

a brain to be embodied?’” (Wolve 1999)

experiences (Sagiv and Ward 2006). Recent neurosci-

The difficulty of defining a kinesthetic sense

entific research reports synesthetic experiences with

coherently might also be owed to the fact that it is,

non-perceptual, semantic inducers like words,

though added to the interoceptive sensory domain,

letters, numbers, time-units, musical notes, personali-

so obviously involved in all sensory experiences

ties, emotions and even moving styles of swimming.

and that it is therefore difficult to discriminate it

The swimming-style synesthesia here is of particular

from the exteroceptive sensory domains. When we

interest, as it has been shown that not only watching,

“see with the legs” (Foerster 1977), move our eyes to

but the mere imagination of people swimming in

be able to see (Spering and Carrasco 2015), touch

different styles evokes synesthetic color experiences

non-existent surfaces with our fingers (Robles-De-

in probands (Nikolić et al. 2011, Rothen et al. 2013).

La-Torre 2006), then kinesthesia is always part of

Ansorge et al. have shown for instance that the

it. It is integrating, and literally incorporating, all

“spatial meaning of an unconscious visual word,

senses in a synesthetic way, so to speak. This is

such as up, influences the perceived location of a

the case not only regarding the multi- and cross-

subsequently presented auditory target.” In a study

modal condition of synesthesia, but also regarding

they found out that unconscious processing “extends

its semantic, and even social, co-condition.

to semantic-sensory connections between different modalities” and contextualized these findings among

Spatiotemporal Concepts in Synesthetic Perception

others within the concept of embodied cognition (Ansorge et al. 2015).

As I have said before, the kinesthetic sense not only

Thus, spatiotemporal parameters of move-

integrates the other senses, but moreover – on the

ment seem to play an important role in the pro-

semantic side of perception – it also incorporates

cessing of the (syn)aesthetic perception of colors,

concepts of time and space, both of which in the

sounds, and numbers. Ward et al. have shown in

context of synesthesia research have been proven

a unique and large-scale interdisciplinary study

to be fundamentally involved in synesthetic

on synesthetic perception, that “a general tendency

perception. Mroczko-Wąsowicz and Nikolić have

for the visual experiences to move in a left-to

developed the model of ideasthesia to explain

right direction, ...” occurs (Ward et al. 2008). The

this phenomenon of the semantic origins of

study’s authors suggest that this phenomenon

synesthesia. It is “driven by semantic mechanisms

could have two possible causes: “First of all, there

as a part of mental representations whereby

is a general bias in attention to the left of space

each synesthete’s individual semantic network

that may reflect right hemispheric specialization

contains concurrents as a part of the meaning

for spatial processes ... Second, it may reflect the

of the inducing stimuli.” (Mroczko-Wąsowicz

cultural tendency for left-to-right reading.” (Ward

and Nikolić 2014)

et al. 2008)

17

Another strong cross-modality between sensory and

body schemas in language which build on spatial

spatial perception has been demonstrated by Cuskley

and motional body concepts (Lakoff and Johnson

with the example of sound and motion synesthesia:

1980/2003). Such “spatial primitive image schemas”

“… sound and motion perception are fundamentally

occur in every language in the world. They provide

connected via low-level multi-modal representations …

information about the locality and the spatiotemporal

motion perception is fundamentally multi-modal, and

context of the (speaking) body and contain spatial and

research in motion perception has shown that audi-

temporal deictic (here, there, after etc.) and directive

tory information in particular is vital to the perception

(into, up, forward etc.) indicators, container meta-

of motion” (Cuskley 2013). This strong connection

phors (in, through, out etc.), statements about the

between motion and sound in speech perception

constitution of something (part - whole), about the re-

is described by the author as a “shared cross-modal

lation of objects (in front of, faster than, etc.), the ver-

association between linguistic sound and motion”

ticality and horizontality of objects, and many more

(Cuskley 2013).

(cf. Lakoff 2015). These image schemas are genuinely

features (and of visual forms) has also been shown

an important role in every language and, moreover, in

amongst others with the well-known example of the

the context at hand, because they are the basis for the

takete- and maluma-called angular shapes

relational connections of space, time, conceptual and

(Ramachandran and Hubbard 2003); often also

social features of digital artworks, as I will show fur-

referred to as kiki and bouba, cf. for instance Danko

ther below. In deictic words the semantic meaning is

Nikolić’s article in this book. Beside spatial parameters,

fixed, but dependent on the context. Words like here

temporal parameters have also been under explo-

and there, up and down, now and tomorrow, me and

ration – for instance the circadian (or daily) rhythm

you provide information about the spatiotemporal

which seems to rely on some kind of chronoception

and social contexts of an utterance. They serve as

or “time sense” we have (Rao et al. 2001) – which

indicators of kinesthetic features in verbal expressions

indicates a strong involvement of kinesthetic parame-

but also can be expressed by non-verbal media like

ters in synesthetic experiences.

images or sounds. For instance, a sound can be

Kinesthesia therefore serves as a good example of

perceived as coming “from above” and therefore

the multimodal constitution of all sensory domains.

unconsciously have an impact on the perception of

It can be observed as overall existent across all other

visual objects (Ansorge et al. 2015).

sensory modalities – no matter if vision, audio, smell, taste, touch or the proprioceptive senses, kinesthetic

Furthermore, language can express metaphorical structures between different fields of semantic or

features of motion, direction, position, speed, rhythm,

sensory domains. The cross-modal correlations

acceleration etc. can be found in seeing, hearing,

between different sensory domains have been shown

smelling, tasting, touching and feeling. These consid-

in a number of famous studies (Marks 1974, Marks

erations are also backed by theories of language

1982, Marks 1996, Marks 1997, Yeshayahu and Gil

embodiment and embodiment of media, which basi-

2008). Metaphors are genuinely kinesthetic figures.

cally state that all perception and cognition in animals

Their fundamental structure is the transfer (the

is per se mediatized through bodily experience.

mapping). In metaphors, the features or aspects of

So, if we speak of synesthesia as the merging of

one issue/object/situation (source domain) are

the senses, then we can speak of kinesthesia as the

transferred to (or mapped onto) another issue/object/

integration of sensory, semantic and social perception.

situation (target domain). This transfer takes place from one (sensory or semantic) domain to another, as

Kinesthetics in Language

for instance the expression “bitter cold” transfers a

In the context of kinesthesia as the integrating sense,

gustatory trait to the kinesthetic (thermoceptive) do-

one must bring up Lakoff and Johnson’s theory of

main. In the expression “a piercing glance” a feature of

THEORIES ON DIGITAL SYNESTHESIA

18

Katharina Gsöllpointner

kinesthetic. Amongst them deictic expressions play

Syn-Aesthetics of Digital Art

The connectedness of vocal sound and kinesthetic

the source domain “kinesthetic” (to pierce) is

in the absence of a suitable content” (Porat and Shen

transferred onto the target domain “vision” (transfer

2015, 79). In other words: a mere form can enhance a

from the kinesthetic/haptic domain to the visual

metaphor or a metaphorical meaning respectively.

domain by means of the image schema “container”).

If we consider this to be also true for non-language

So, the kinetic aspect of transfer is inherent to a

media such as image, sound, taste, smell, haptics, and

metaphor. But metaphorical transfers of domain

kinesthetics such modal features of other media forms

features also take place in non-verbal domains, such

must be detectable by describing and categorizing

as in vision, audio and the other sensory domains.

them and investigating their metaphorical possibilities. These could for instance be kinesthetic metaphors in

Media Forms of “Imposed Metaphoricity”

all sensory domains.

It is not only metaphors themselves that are kin-

highly constrained, and is restricted almost exclusively

Although the “use of MIF in natural language is

esthetic figures, however. In language, for instance,

to metaphorical contexts”, (Porat and Shen 2015) it

forms can even be found which merely impose

would be of great interest to match these results onto

metaphorical transfers. Porat and Shen have intro-

non-verbal media in the sense that certain media

duced such a concept of “Imposed Metaphoricity” in

forms in digital artworks (MIF language forms in Porat

language (Porat and Shen 2015) to provide a model

and Shen’s study) might basically impose cross-modal

for a phenomenon “in which metaphorical processing

visual, auditory, gustatory, olfactory, haptic/tactile,

is imposed on a given expression regardless of its

and kinesthetic metaphorical processing as well. From

semantic content” (Porat and Shen 2015). They

this subsumption questions like the following arise:

suggest that besides standard language, metaphorical

How exactly do non-verbal MIF in various sensory

forms like nominal metaphors (X is Y) and similes

domains (vision, audio, taste, smell, touch, kines-

(X is like Y), so-called “metaphoricity-inducing forms”

thetics) appear and what would these aesthetics look

(MIF) are used in order to “impose metaphorical

like? How could non-verbal MIF be compared to the

processing on any two nouns, regardless of semantic

MIF in language? How could the imposing function

factors such as constituent meaning or the context of

of non-verbal MIF be described? What would visual,

the expression. This group includes various linguistic

auditory, gustatory, olfactory, haptic/tactile, and

constructions that are commonly used to express

kinesthetic MIF look like in digital art? Can semantics

intensification, such as nominal sentences with ad-

also serve as conceptual MIF? Such future research

verbial intensifiers (this X is such a Y, this X is really Y),

would provide useful information about the role of

appositive genitive constructions (this is a Y of an X),

multimodal media aesthetics in digital art in regard to

and question-like exclamations (what a Y this X is)”

synesthetic perception.

(Porat and Shen 2015). As I have stated, metaphors transfer (map) features from one semantic (or sensory) domain to another (source to target domain) to make certain characteristics of objects/issues explicit, and therefore have a kinetic/motional part within their structure. However, Porat and Shen’s findings suggest moreover that the metaphorical processing itself can be activated by two routes: the “semantic (lexical, contextual)” and the “structural (or form dependent)”. “Hence, although the process of assigning metaphorical interpretation can be described as a manipulation of semantics, it is not necessarily derived from it, and can be triggered even

19

Kinesthetics in Digital Art Digital art, due to its above mentioned algorithmic,

rhythm, which refers to the chronoceptive as well

cross- and multimedia basis, clearly has a strong

as to the auditory and visual domains.

spatiotemporal structure. Thus its aesthetics always,

Third, and this is what is interesting in the con-

and from the very beginning onward, contain funda-

text at hand, digital artworks by their technological

mental features of motion in terms of the alteration

conditions are basically compounded of kinesthetic

of space and time for an object or body. These

modalities through all sensory and semantic domains.

spatiotemporal shifts take place in the form of aes-

By the example of the 15 artworks which have been

thetic media transformations, either within the same

accomplished by artists in the course of the DIGITAL

sensory modality, or from one sensory modality to

SYNESTHESIA research process, Fig. 6 demonstrates

another. Additionally digital art — as any other art — 

how modalities in various sensory domains can be

always transports a semantic field, in which contents

transferred to and from kinesthetic features of digital

of “meaning” are cached, stacked or openly narrated.

artworks.

If we consider kinesthetics as the sensory domain

basically responsible for synesthetic experiences, then

the work. Detecting verbal indicators in the titles

we must look at how kinesthetic features in media

and their linguistic function shows that 13 out of 15

forms of digital artworks accomplish this task. For this

artworks explicitly have a kinesthetic indicator in

I suggest a procedure of analyzing media forms in

their title. Only Data Music (Fig. 1) and Mirrors of the

artworks which indicate trans-modal kinesthetic

Unseen (MotU #4 – #6) (Fig. 7) do not have an explicit

features in other sensory domains as well as in the

kinesthetic indicator, although music and mirror both

“semantic field”. Before taking this step I would

strongly point to temporal and spatial aspects of

like to differentiate between three groups of art-

sound and vision.

works in which the possible analysis of kinesthetic

2. Describing the primary sensory modalities/do-

cross-modal features basically can be undertaken:

mains which are addressed by each of the artworks, it

First there is the group of artworks which are

is also obvious that in all of them the kinesthetic sense

grounded on motion media and are not necessarily

is either top-listed or among the first three modalities;

digital. In this group I count film, video, performance,

only two pieces have other senses addressed more

dance, music/sound. Second, in static, non-dynamic

obviously: although in E.E.G. KISS (Fig. 8) kinesthetics

artworks many kinesthetic modalities can be detected,

verifiably has a strong connectedness with the work’s

as for instance in paintings, sculptures or literature,

focus on touch and taste; and in Data Music (Fig. 1)

and all of them can comprise of metaphorical kines-

the playing of the “stave-strings” also has an obvious

thetic features, like a fast brush in a painting or a

kinesthetic aspect beside the strong touch aspect.

heavy piece of literature. Further spatial-kinesthetic

3. The semantic field of an artwork is of course

modalities in paintings or objects would be

dependent on the visitor’s /user’s individual and

features like right, left, high, far, wide, narrow, low,

cultural background. Nevertheless, certain formal

deep, balanced, heavy, light. Temporal-kinesthetic

indicators (such as, for instance, the title or the con-

modalities such as fast, slow, accelerating, speedy

tents of images or songs, the material that objects

might also be found in static artworks in regard to

and installations are made of etc.) provide information

the cross-modal correlations of sensory features

about the narrative field which encompasses the

(Marks 1974, Marks 1982, Marks 1996, Marks 1997,

artwork. Therefore the table gives one example of

Yeshayahu and Gil 2008). And one motional-

a possible semantic field for each of the artworks.

kinesthetic modality in static works definitely is

Most of these semantic fields also are strongly

THEORIES ON DIGITAL SYNESTHESIA

20

Katharina Gsöllpointner

1. The artwork title gives a good impression of the primary sensory domains which are addressed by

Syn-Aesthetics of Digital Art

which integrates sensory with semantic perception via its holistic bodily condition and which therefore is

1. Artwork Title Verbal Indicators of Kinesthesia

2. Primary Sensory Modalities / Domains

3. Exemplary Semantic Field (content of text if included as artistic medium)

4. Exemplary Kinesthetic Verbal Expressions and Metaphors

5. Trans-modal Media Aesthetics (sensory directionality)

At Play at (deictic preposition)

kinesthetics, spatioception (proprioception, touch, balance, audio, vision)

a playground

playing around

images are projected onto boxes; boxes become images (vision < > kinesthetics)

Bestiary for the Minds of the 21st Century: Genomic Opera 21st century (temporal indication)

kinesthetics, touch, vision, audio

multimedia hybrids

turning the page

transference of sound data into 3D-print into image data etc. (kinesthetics > audio and >vision)

Data Music —

touch, audio, vision

a musical concert

playing strings on one’s heart

sound is produced by moving/ touching the “stave-strings” (kinesthetics > audio)

Diver diving (act of body movement, associated with ‘down’)

kinesthetics, vision, audio

a dive

diving into sth.

an image turning upside down (vision > kinesthetics)

E.E.G. KISS (brain) waves (form of motion), kissing (act of tasting, touching & body movement)

taste, touch, vision, audio

a kiss

being touched by sth.

images of (brain) waves (kinesthetics > vision )

facades facade (haptic & spatial indicator)

vision, audio, kinesthetics (balance, visceroception)

street view content of the text

walking the streets

low tones of speech (kinesthetics > audio)

The Flying Umbrella Project flying (act of body movement, associated with ‘up’)

kinesthetics, touch, audio

flying umbrellas

on the fly

height equivalents in sound (a high tone) (kinesthetics > audio)

I am Sound I (deictic noun)

vision, audio, kinesthetics

a selfie content of the text

looking ahead

an image becomes the vibrating membrane of a loudspeaker (audio > vision > and kinesthetics)

IN_SIDE VIEW in (deictic preposition), side (spatial indication)

taste, touch, vision, kinesthetics

a walk through Angkor Wat 

on the tip of the tongue

a tongue switch device for images (taste > kinesthetics and > vision)

MotU #4 – #6 (Mirrors of the Unseen) —

vision, kinesthetics

after-image, blind spot, synesthesia content of the text

blind spot

after-image (kinesthetics > vision)

Sound Calligraphy calligraphy (act of body movement)

vision, audio, kinesthetics 

writing a text content of the text

drawing a circle

handwriting is sound (vision > audio)

Space Time space, time

kinesthetics, chronoception, audio 

clocks, atomic time

time is drifting

a clock is loud (kinesthetics > audio)

Topography of Movement topography (spatial indicator), movement (motion indicator)

kinesthetics, vision, audio

a topographical space

body-scan

a light projection is scanning a wall (kinesthetics > vision)

Transmission+Interference transmission, interference (both spatiotemporal processes)

vision, audio, kinesthetics 

a transmission with an interference content of the text

to mess something up

light is sound (vision > audio)

VERTICAL 2 vertical (body schema)

kinesthetics (balance, visceroception), audio, vision

an elevator

feeling up and down

high pitches of sound (kinesthetics < > audio)

Fig. 6

21

kinesthetically defined topics, such as e.g., a play-

process of sound, object, image and motion data

ground (Fig. 4), a musical concert (Fig. 1), a dive (Fig. 9),

transfer into each other and back again as its aesthetic

a hand tapping over walls (Fig. 10), flying umbrellas

basis. Such trans- and intermodal metaphors between

(Fig. 11), a walk in Angkor Wat (Fig. 3), writing calli-

kinesthesia and the other senses can also be detected

graphies (Fig. 12), the fingers of ticking clocks (Fig. 13),

by describing strictly formal parts of the artwork. For

etc. Additionally, in those artworks which use text as

instance in Diver (Fig. 9) images are turned upside

part of their conception, the content of the respective

down. Here a condition from the kinesthesia domain

text also represents a semantic field. So, for instance

(“upside down”) is projected onto the vision domain

in Ruth Schnell’s MotU #4 – #6 (Fig. 7), the words

(“image”). The image becomes kinesthetic, not only

appearing from the light sticks form a “narration” on

because it is — as any film or video — a moving picture

synesthesia, as does the text source (digital protocols

(> “turned”), but mainly because it is turned upside

of ASCII and frequency conversion) respectively in

down. Another striking and well known example

Transmission+Interference (Fig. 14) by David Strang

of the inter-modality of sensory features is the corre-

and Vincent Van Uffelen. In Ruth Schnell’s facades

lation in the perception of pitch (auditory domain)

(Fig. 15) the voice which reads out a text by Friedrich

and height (kinesthetic domain), which is used to

Kittler about the NSA creates a semantic field in

model a spatial experience by acoustic forms in the

regard to topics of secret services and interception.

artwork VERTICAL 2 (Fig. 17).

Additionally, it must be stressed that the semantic field is not only created by the text’s content alone,

This exemplary description of kinesthetic media fea-

but with the same impact by its media-reflective

tures in the DIGITAL SYNESTHESIA artworks is a first

aesthetics; in this case the whispering mode of

step toward exploring the suggested fundamental

the voice.

role of kinesthesia in the integration of sensory and

metaphors stem from the context of the semantic

by means of media aesthetics. It is evident that kin-

field of each artwork. They thus show strong kines-

esthetic features in digital art are an important part

thetic features, mainly demonstrated by either deictic

of both the conception and the perception of the

expressions (at, around, into, on top, up, down etc.),

artworks. Artworks here serve once more as best

and/or by words indicating motion (diving, walking,

practice examples of synesthesia research because

turning, playing, drawing, writing etc.). Such associa-

they are not just used as objects for the observation

tions might for instance be related to a linguistic

of physiological and psychological processes in art

metaphor like “on the tip of the tongue” in IN_SIDE

observers, but serve as research methods about per-

VIEW (Fig. 3), “on the fly” in The Flying Umbrella

ception themselves. They provide information about

Project (Fig. 11), or “playing strings on one’s heart”

the intertwinedness of media, form and perception by

in Data Music (Fig. 1); or they might be related to

means of their media aesthetics.

ambiguous language expressions like “turning the

For future detailed investigations about kinesthesia

page” in Bestiary for the Minds of the 21st Century

in digital art, a number of research questions needs

(Fig. 16), “looking ahead” in I am Sound (Fig. 2), or

to be explored further in detail: How exactly are modal

“being touched by something” in E.E.G. KISS (Fig. 8).

features in digital artworks perceived by visitors/

5. Trans-modal media aesthetics can be described

users? What do users perceive and sense exactly in

as the aesthetic media forms of the artwork which

their bodies and bodyparts, when they interact with

suggest the transfer of either kinesthetic feature into

digital artworks? What are the functional differences

another sensory domain, or vice versa. Bestiary for

between static and dynamic features in regard to the

the Minds of the 21st Century (Fig. 16), for instance,

process of perception? Do dynamic media “double”

does this in an extremely explicit way as it uses the

kinesthetic perception parameters in the visitors/

concept of the almost endless transformation

users by means of their motional traits? And: What

THEORIES ON DIGITAL SYNESTHESIA

Katharina Gsöllpointner

semantic components in (synesthetic) perception

22

Syn-Aesthetics of Digital Art

4. Exemplary kinesthetic verbal expressions and

role does social interaction among the audience

could also be considered as the “sense of empathy”.

have for the aesthetics of the work?

The ability to change perspectives — literally and metaphorically — would then only be possible

The last issue especially refers to Alain Berthoz’ statement that gestures as kinesthetic forms are

on the basis of kinesthesia (cf. Berthoz 2014).

“synchronizing the time of two individuals”

These social or “gesture” features of digital art are

(Berthoz 2014). According to this idea, kinesthesia

another form aspect for the aesthetic condition

would not only be the sense which integrates

of DIGITAL SYNESTHESIA which surely deserve

the other senses, semantics, time and space, but

more attention.

Fig. 7

Fig. 6 Kinesthetic features, media and metaphors in DIGITAL SYNESTHESIA artworks

Fig. 7 Ruth Schnell: MotU #4 – #6, 2016 (Exhibition View at AIL Vienna) Photo: Birgit and Peter Kainz

Fig. 8 Karen Lancel /Hermen Maat: E.E.G. KISS, 2016 (Performance at AIL Vienna) Photo: Birgit and Peter Kainz Fig. 8

23

Katharina Gsöllpointner

Fig. 9

Syn-Aesthetics of Digital Art

Fig. 10

Fig. 11

THEORIES ON DIGITAL SYNESTHESIA

24

Fig. 9 kondition pluriel: Diver (Sketch), 2016. © kondition pluriel

Fig. 10 Ruth Schnell: Topography of Movement, 2016 (Exhibition view AIL Vienna) Photo: Birgit and Peter Kainz

Fig. 11 Alan Kwan: The Flying Umbrella Project, 2016 (Performance at AIL Vienna) Photo: Birgit and Peter Kainz

Fig. 12 Ulla Rauter: Sound Calligraphy/ Performance, 2015-2016 Photo: Ulla Rauter Fig. 12

Fig. 13 Karl Heinz Jeron: Space Time, 2016 Photo: Karl Heinz Jeron

Fig. 13

25

Fig. 15

THEORIES ON DIGITAL SYNESTHESIA

Syn-Aesthetics of Digital Art

Katharina Gsöllpointner

Fig. 14

26

Fig. 16

Fig. 14 David Strang / Vincent Van Uffelen: Transmission+ Interference, 2016 (Exhibtion View at AIL Vienna) Photo: Birgit and Peter Kainz Fig. 15 Ruth Schnell: facades, 2016 © Ruth Schnell Photo: Johnny Ranger

Fig. 16 Marcello Mercado: Bestiary for the Minds of the 21st Century: Genomic Opera, 2016 (Exhibtion View at AIL Vienna) Photo: Birgit and Peter Kainz

Fig. 17

Fig. 17 Anke Eckardt: VERTICAL 2 (Sketch), 2016 Digital drawing: Anke Eckardt

27

Amin, Maina, et al. 2011. “Understanding Grapheme Personification: A Social Synaesthesia?” Journal of Neuropsychology 5: 255–82. Banissy, Michael J. and Jamie Ward. 2007. “Mirror-touch Synesthesia is Linked with Empathy.” Nature Neuroscience 10: 815–17. Berthoz, Alain. 2000. The Brain’s Sense of Movement. Cambridge, MA: Harvard University Press. Berthoz, Alain. 2014 “Emotional sharing with the artwork: gesture, gaze, empathy.“ International Workshop of the French National Research Agency (ANR) Project Visual Art and Emotion, May 20, 2014: Accessed January 4, 2016. https://www.youtube.com/ watch?v=fmiNzwIngY0 Cuskley, Christine. 2013. “Mappings between linguistic sound and motion.“ PJOS 5(1). 39–62. Evers, Frans. 2012. The Academy of the Senses: Synesthetics in Science, Art, and Education. The Hague: ArtScience Interfaculty Press. Fingerhut, Joerg. 2011. “Synaesthesia and Kinaesthetics. An Introduction.“ Habitus in Habitat III. 8, eds. Fingerhut J, Flach S and Söffner J. 7–19. Foerster, Heinz von and Moshe Feldenkrais. 1977. “A Conversation.” eds. Leri and Sutherland (transcription), San Fransisco. Fulkerson, Matthew. 2014. “Rethinking the senses and their interactions: the case for sensorypluralism.” Frontiers in Psychology. Gsöllpointner, Katharina. 2015. “Digital Synesthesia. The Merge of Perceiving and Conceiving.” Cyborgian Images. The Moving Image between Apparatus and Body. Yearbook of the Moving Image Studies Vol. 1, eds. Grabbe L, Rupert-Kruse P and Schmitz N. Darmstadt: Büchner Verlag. 108–133.

Hale, Joanna, et al. 2014. “Better together? The Cognitive Advantages of Synaesthesia for Time, Numbers, and Space.” Cognitive Neuropsychology. Lakoff, George. 1987. “Position Paper on Metaphor.“ ACL Anthology A Digital Archive of Research Papers in Computational Linguistics. Accessed November 22, 2015. http://www. aclweb.org/anthology/T87-1039 Lakoff, George. 2015. “How Brains Think: The Embodiment Hypothesis“. Keynote Address recorded March 14, 2015 at the inaugural, International Convention of Psychological Science in Amsterdam. Accessed December 02, 2015. https://www.youtube.com/ watch?v=WuUnMCq-ARQ Lakoff, George, and Mark Johnson. 1980/2003. Metaphors We Live By. Chicago: University of Chicago Press. Marks, Lawrence. 1974. “On Associations of Light and Sound: The Mediation of Brightness, Pitch and Loudness.” American Journal of Psychology 7: 173–88. Marks, Lawrence. 1982. “Bright sneezes and dark coughs, loud sunlight and soft moonlight.“ Journal of Experimental Psychology: Human Perception and Performance, Vol 8(2): 177–193. Marks, Lawrence. 1996. “On Perceptual Metaphors.” Metaphor and Symbolic Activity 11: 39–66. Marks, Lawrence. 1997 (1975). “On Colored-hearing Synesthesia: Crossmodal Translations of Sensory Dimensions.” Synaesthesia: Classic and Contemporary Readings, eds. Simon Baron-Cohen and Harrison John, 49–99. Cambridge, MA: Blackwell. Marks, Lawrence, and Catherine Mulvenna. 2013. “Synesthesia, at and near its borders.” Frontiers in Psychology 4: 1–4. Martino, Gail and Lawrence Marks. 2001. “Synesthesia: Strong and Weak.” Current Directions in Psychological Science 10: 61–65.

THEORIES ON DIGITAL SYNESTHESIA

Merriam Webster Dictionary. 2015. Kinesthesia. Accessed June 30, 2015. http://www.merriam-webster.com/ dictionary/kinesthesia. Mroczko-Wąsowicz, Aleksandra and Danko Nikolić. 2014. “Semantic Mechanisms May Be Responsible for Developing Synesthesia.” Frontiers in Human Neuroscience 8, 509: 1–13. Nikolić, Danko, et al. 2011. “Swimming-style Synesthesia.” Cortex 47: 874–79.

Wolve, Charles T. 1999. “Proprioception”. Journal of Neuro-Aesthetic Theory 1. Accessed January 13, 2016. http://www.artbrain.org/category/ journal-neuro-aesthetic-theory/ journal-neuro-aesthetic-theory-1/ Yeshayahu, Shen, and Gil David 2008. “Sweet Fragrances from Indonesia: A universal principle governing directionality in synaesthetic metaphors.” New Beginnings in Literary Studies. Newcastle: Cambridge Scholars Publishing, 49–71.

O’Regan, Kevin and Alva Noë. 2001. “A sensorimotor account of vision and visual consciousness.” Behavioral and Brain Sciences 24. 939 –1031. Porat, Roy, and Shen Yeshayahu. 2015. “Imposed Metaphoricity.“ Metaphor and Symbol 30: 77–94. Proske, Uwe, and Simon Gandevia. 2009. “The Kinaesthetic Senses.” The Journal of Physiology 587, 17: 4139–46. Ramachandran, V.S. and Edward M. Hubbard. 2003. “The Phenomenology of Synesthesia.” Journal of Consciousness Studies 10: 49–57. Rao, Stephen M., et al. 2001. “The Evolution of Brain Activation during Temporal Processing.” Nature Neuroscience 4: 317–23. Robles-De-La-Torre, G. 2006. “The importance of the sense of touch in virtual and real environments.“ MultiMedia, IEEE 13/3. 24-30. doi:10.1109/ MMUL.2006.69 Rothen, Nicolas, et al. 2013. “Psychophysiological Evidence for the Genuineness of Swimming-style Colour Synesthesia.” Conscious. Cogn. 22: 35–46. Sagiv, Noam, and Jamie Ward. 2006. “Crossmodal Interactions: Lessons from Synesthesia.” Progress in Brain Research 155: 259–71. Spering, Miriam, and Carrasco Marisa. 2015. “Acting without seeing: eye movements reveal visual processing without awareness.“ Trends in Neurosciences 38,4: 247–258. Ward, Jamie, et al. 2008. “The Aesthetic Appeal of Auditory-visual Synaesthetic Perceptions in People Without Synaesthesia.” Perception 37: 1285–96.

28

Katharina Gsöllpointner

Ansorge, Ulrich, et al. 2015. “Unconscious Cross-Modal Priming of Auditory Sound Localization by Visual Words.“ Journal of Experimental Psychology: Learning, Memory, and Cognition.

Gsöllpointer, Katharina and Sibylle Moser. 2009. “Sprache Beobachten.” Sprachmedien. Accessed January 19, 2016. http://www. sprachmedien.at/downloads/ sprachbeobachtung_loop.pdf.

Syn-Aesthetics of Digital Art

References

Synesthesia: Perception, Language , Digital Art

Lawrence E. Marks / Catherine M. Mulvenna

 Keywords   crossmodal correspondence, synesthesia, perception, language, metaphor, idiographic, nomothetic, digital art

“I was of three minds,

The Digital Synesthesia project “is a transdisciplinary

Like a tree

artistic research area which explores the (syn-)aes-

In which there are three blackbirds.”

thetics of digital art. It especially focuses on its technological, aesthetic and media conditions

(Wallace Stevens: Thirteen Ways of Looking at a Blackbird)

which make it possible to provide synesthetic experiences …” (Digital Synesthesia 2014). Central to the project is the question: How might digital art activate synesthesia in individuals who may not ordinarily experience it? To this we add a second, related question: How might digital art evoke experiences that parallel synesthetic experiences, while nevertheless differing in their origin and phenomenology? Consequently, this article explores issues we deem central to the relations among digital art, borderline synesthesia-like experiences (Marks and Mulvenna 2013) and synesthesia itself (Marks 2014a).

29

Synesthesia: Idiographic and Nomothetic Approaches We start by considering two complementary ap-

his entire tongue ...” (Luria 1968, 45) – the flavors of

proaches to the study of human behavior that can help

foods such as borscht being, of course, specific to

in understanding and sorting out the many facets of

particular cultures.

approaches focus on those individual characteristics or

what makes every synesthete’s experiences unique,

traits that make people unique. In contrast, nomothetic

so too does it encompass what synesthetes share –

approaches focus on characteristics that people share:

and thereby calls for a complementary, nomothetic

characteristics that are widespread, perhaps even

approach. Considerable work has focused on a

universal. Art, whether digital or not, emerges from an

panoply of perceptual correspondences, in both

artist’s imagination, which can creatively tap both what

synesthetes and non-synesthetes, that are universal.

is individual and what is universal.

Despite the individuality of synesthesia in its full,

Synesthesia refers to the evocation of sensations or

nuanced detail, the study of synesthesia also reveals

perceptions in a secondary domain through stimula-

general rules and regularities, unity amidst diversity

tion that typically evokes experiences only in a primary

in patterns of perception: A notable rule, first

domain. The domains may comprise different sensory

suggested more than a century ago (Bleuler and

modalities, as when sounds evoke visual shapes, or

Lehmann 1881), is the correspondence between

different domains of the same modality, as when printed

pitch and brightness across the otherwise diverse

numbers evoke colors (e.g., Mulvenna and Walsh 2005).

instances of sound-to-sight synesthesia (e.g.,

It is perhaps a truism to say that every synesthetic

Karwoski and Odbert 1938; for review, see Marks

perceiver, or synesthete – like every snowflake – is

2014a). Synesthetes may disagree, for example, on

unique; for the precise shapes, hues, and flavors that

the specific colors evoked by a piano note in C, but

a synesthete experiences, together with the precise

nevertheless agree that the color’s brightness

stimuli that trigger these experiences, are an individ-

increases as the note is played in increasingly higher

ual matter. Moreover, each synesthete’s experiences

octaves. Auditory-visual synesthesia in particular

are embedded within her or his culture and language.

reveals nomothetic principles: Synesthetic visual

Consider the Russian synesthete S., who, responding to

images commonly decrease in size and increase in

a loud, low pitched tone, “experienced a flavor like that

brightness as pitch increases, but increase in size as

of sweet and sour borscht, a sensation that gripped

loudness increases.

Crossmodal Correspondences in Non-synesthetes Overt Correspondences in Perception

na 2012). Indeed, several universal correspondences

Analogous crossmodal correspondences also charac-

revealed in auditory-visual synesthesia, including the

terize perception in non-synesthetes, to whom, for

three described above, have analogs in the perceptual

example, colors of greater brightness align with

experiences of non-synesthetes: pitch with brightness,

sounds of higher pitch (Wicker 1969; Marks 1974), as

pitch with size (inversely), and loudness with size (for

Wundt (1874) long ago noted. Further, the correspond-

reviews, see Spence 2011; Marks 2014b).

ence between pitch and brightness not only pervades Western cultures but also appears in the crossmodal

Overt Correspondences in Language

perception of the Himba, denizens of preliterate, no-

Importantly, correspondences between dimensions of

madic tribes from the deserts of northwest Namibia

different modalities or domains also emerge when the

who have little contact with Western culture (Mulven-

stimuli are words that describe or signify perceptual

THEORIES ON DIGITAL SYNESTHESIA

30

Lawrence E. Marks / Catherine M. Mulvenna

Just as the study of synesthesia encompasses

Synesthesia: Perception, Language, Digital Art

synesthesia: idiographic and nomothetic. Idiographic

experiences, rather than sensory stimuli that directly

correspondences perhaps develop through experi-

arouse the experiences. The words SOFT and LOUD

ence with statistical regularities in the world (Parise

imply dim and bright color sensations indirectly, much

2016), while others may reflect non-arbitrary, iconic

as soft and loud sounds do more directly. Similarly,

relations intrinsic to shared meanings (Dingemanse

the words BROWN and YELLOW imply low and high

et al. 2015).

pitch sensations, just as the dark color brown and the Fig. 1 characterizes the correspondence between

Covert Correspondences in Perception and Language

auditory pitch and visual brightness as revealed in the

The examples just described show how overt percep-

literal and connotative meanings of words referring

tual judgments provide simple, explicit, and direct

bright color yellow do more directly (Marks 1982).

to auditory and visual experiences. The words BLACK

evidence of crossmodal correspondence – e.g.,

and BROWN are both darker than YELLOW and WHITE

between visual color and musical pitch. In contrast,

and higher in pitch, whereas COUGH and DRUM

covert measures provide evidence of implicit pro-

NOTE are lower in pitch than SNEEZE and VIOLIN

cessing of correspondences, in both synesthetes and

NOTE, and also darker.

non-synesthetes. A conventional covert measure

With increasing age and cognitive development,

capitalizes on the so-called Stroop effect (Dyer 1973),

crossmodal correspondences become increasingly

which originally revealed how the ability to quickly

available to language. As children learn the direct,

name a printed word’s color (say, green) is impeded

denotative meanings of sensory words, for instance

when the word spells the name (say, ORANGE) of a

that COUGH and SNEEZE refer to acoustically

conflicting color.

different events, children also come to understand

Crossmodal versions of the Stroop paradigm

the indirect, connotative meanings: that COUGHS

show, for example, that people are faster and more

and SNEEZES, like coughs and sneezes, imply different

accurate in identifying a low-pitched tone presented

levels of lightness or brightness. Fig. 2 shows the

with a dark color, and faster in identifying a high-

results of a study that assessed three auditory-visual

pitched tone presented with a bright color – and,

correspondences in children varying in age from 3 to

similarly, identifying dark and bright colors in the

13 years. The study used both a sensory-perceptual

presence of low- and high-pitched sounds (Marks

task, where the children matched sounds and lights,

1987; Melara 1989). Interactions in speed and accuracy

such as tones differing in frequency (pitch) and lights

of response occur for the most part automatically,

differing in luminance (brightness), and a verbal-met-

and appear across many sensory modalities (e.g.,

aphorical task, where the children interpreted brief

vision and hearing) and dimensions (e.g., brightness

crossmodal metaphors, such as SOFT SUNLIGHT and

and pitch) (for reviews, see Marks 2004; Spence 2011).

LOUD MOONLIGHT. The results reveal systematic but distinctive increases with age in “recognizing” all

Importantly, interactions are similar when they arise from stimuli that evoke perceptual qualities,

three correspondences – pitch with brightness,

such as colors and pitches, and when they arise

loudness with brightness, and pitch with size – in

from words that represent perceptual qualities, such

both perception and in language. Perceptual per-

as WHITE and BLACK (Martino and Marks 1999;

formance exceeded verbal performance at all ages,

also Walker and Smith 1985). Perhaps crossmodal

consistent with the view that the correspondences

correspondences in perception and in language

originate in perception (some perhaps during infancy:

activate overlapping mechanisms, tapping the

Lewkowicz and Turkewitz 1980; Walker et al. 2010;

same underlying neural and mental representations

Haryu and Kajikawa 2012), and then become available

when perceptual stimuli evoke sensory qualities

to semantic representation; and semantic representa-

directly and when words evoke sensory qualities

tion may also play an important role in synesthesia

indirectly (see also Evers 2012; Mroczko-Wąsowicz

itself (Mroczko-Wąsowicz and Nikolić 2014). Some

and Nikolić 2014).

31

And what, then, may be the responses of non-

make crossmodal or crossdomain relations directly

synesthetic individuals to works of art, especially

available to spectators. With works relegated to a

including multimedia art? To the extent that works of

single medium such as music or visual art, however,

art generated through digital technologies activate

the crossmodal implications will be far less transpar-

synesthesia or crossmodal correspondence, the re-

ent to spectators, the range of possibilities far more

sulting experiences of digital synesthesia may engage

open ended – thereby posing a far greater challenge

the idiosyncratic, the universal, or both. Perceptual,

to artists’ creative capabilities.

cognitive, and emotional responses to a given digital

Universals provide touchstones for shared mean-

artwork, like responses to any artwork, depend on

ings and understanding. Multimedia works in par-

both the proclivities of the perceiver and the charac-

ticular provide an opportunity to deploy crossmodal

teristics of the art. Artists creating digital works of art,

correspondences, thereby heightening spectator

like artists working in other media, bring their unique

attention to an undercurrent of universal meanings.

histories, their weft of particulars drawn through a

At the same time, by engaging universal correspond-

warp of universals. Spectators too, when engaging

ences within their works, artists may thereby also

with works of art, bring their own tapestries of par-

sensitize spectators to the possibility of other, novel 

ticulars and universals; for perception transcends

correspondences – of novel crossmodal mean-

reception. Perception, too, is a creative act.

ings. With digital art, as with other art forms, artists

Might digital artworks have a privileged capacity

may seek to avoid both the Scylla of overfamiliar

to evoke synesthesia or synesthesia-like experiences,

universality and the Charybdis of idiosyncratic

even in individuals who are not so predisposed?

unintelligibility, seasoning their works with dashes of

Conceivably. It has been reported, for example, that

both so that spectators may construe the art creatively

synesthesia can be induced in non-synesthetes

through what Coleridge called the “secondary

through hypnosis (Cohen Kadosh et al. 2009; but see

Imagination … co-existing with the conscious Will … 

Anderson et al. 2014). Such unusual conditions aside,

[that] dissolves, diffuses, dissipates, in order to re-

the available evidence – both older (e.g., Howells

create” (Coleridge 1817, 296).

Lawrence E. Marks / Catherine M. Mulvenna

Implications for Digital Art: Digital Synesthesia

1944) and more recent (Bor et al 2014) – suggests that extensive, prolonged exposure to, for example, conjunctions of specific colors with specific sounds Synesthesia: Perception, Language, Digital Art

or letters may eventually produce synesthetic experience – or its simulacrum – in otherwise non-synesthetic adults. Even without the presence of overtly synesthetic perception (i.e., developmental synesthesia: Marks 2014a), however, people respond implicitly to, and hence covertly understand or recognize, many crossdomain correspondences. The advent of digital technologies has effected a sea change in artistic possibilities, including new opportunities to capitalize on covert correspondences. Through multimedia works, for example, artists can establish explicit correlations across auditory, visual, and linguistic experiences, tapping universal correspondences or implying novel, idiosyncratic ones, and can thereby

THEORIES ON DIGITAL SYNESTHESIA

32

Fig. 1 Ratings of pitch plotted against ratings of brightness, evoked by the meanings of words referring to visual and to auditory experiences. Words that refer to brighter colors metaphorically imply higher pitch, and words that refer to higher-pitched sounds metaphorically imply greater brightness. The crossmodal correspondence in language thereby mimics the crossmodal correspondence in perception (based on data from Marks 1982).

Fig. 2 The proportion of children of different ages who associated higher pitch with greater brightness, greater loudness with greater brightness, and higher pitch with smaller size, in tasks of auditory-visual perceptual choice (left panel) and analogous tasks of rating verbal, metaphorical meaning (right panel). Random performance in each task is 50 % (dashed line). In each, the capacity of children to recognize crossmodal correspondences increases with age, performance at a given age being superior in the perceptual task compared to the metaphorical, verbal task (data of Marks, Hammeal, and Bornstein 1987).


 


 
 














 



Fig. 1

 
 


-,,




 '+% '"&&


 '+% '"&&







0/





#("&&+% '"&&




/,


#("&&+
% '"&&


 '+ *


 '+ *









./


"&#%)+%$'(!





































%!+ '$#% !











,














 











Fig. 2

33








 








References Anderson, Hazel P., Anil K. Seth, Zoltan Dienes, et al. 2014. “Can grapheme-color synesthesia be induced by hypnosis?” Frontiers in Human Neuroscience 8:220. Published April 28, 2014. doi. org/10.3389/fnhum.2014.00220 Bleuler, Eugen and Karl Lehmann. 1881. Zwangmässige Lichetempfindungen durch Schall und Verwandte Erscheinungen. Leipzig: Fues’ Verlag. Bor, Daniel, Nicolas Rothen, David J. Schwartzman, et al. 2014. “Adults can be trained to acquire synesthetic experiences.” Scientific Reports 4:7089. Published November 18, 2014. doi.org/10.1038/srep07089 Cohen Kadosh, Roi, Avishai Henik, Andras Catena, et al. 2009. “Induced cross-modal synaesthetic experience without abnormal neuronal connections.” Psychological Science 20: 258–265. Coleridge, Samuel Taylor. 1817. Biographia Literaria; or Biographical Sketches of My Literary Life and Opinions, Volume 1. London: Rest Fenner. Digital Synesthesia. 2014, “ABOUT,” accessed October 14, 2015. http:// digitalsynesthesia.net/wp Dingemanse, Mark, Damián E. Blasi, Gary Lupyan, et al. 2015. “Arbitrariness, iconicity, and systematicity in language.” Trends in Cognitive Sciences 19: 603–615. Dyer, Frank N. 1973. “The Stroop phenomenon and its use in the study of perceptual, cognitive, and response processes.” Memory & Cognition 1: 106–120. Evers, Frans. 2012. The Academy of the Senses: Synesthetics in Science, Art, and Education. The Hague: ArtScience Interfaculty Press. Haryu, Etsuko and Sachiyo Kajikawa. 2012. “Are higher-frequency sounds

Howells, Thomas H. 1944. “The experimental development of color-tone synesthesia.” Journal of Experimental Psychology 34: 87–103.

Marks, Lawrence E., Robin J. Hammeal, and Marc H. Bornstein. 1987. “Perceiving similarity and comprehending metaphor.” Monographs of the Society for Research in Child Development 42: 1–91.

Karwoski, Theodore and Henry S. Odbert. 1938. “Color-music.” Psychological Monographs 50: 1–60.

Marks, Lawrence E. and Catherine M. Mulvenna. 2013. “Synesthesia, at and near its borders.” Frontiers in Psychology 4:651. Published September 26, 2013. doi.org/10.3389/fpsyg.2013.00651

Lewkowicz, David J. and Gerald Turkewitz. 1980. “Cross-modal equivalence in early infancy: Auditory-visual intensity matching.” Development Psychology 16: 597–607.

Martino, Gail and Lawrence E. Marks. 1999. “Perceptual and linguistic interactions in speeded classification: Tests of the semantic coding hypothesis.” Perception 28: 903–923.

Luria, Alexander R. 1968. The Mind of a Mnemonist. New York: Basic Books.

Melara, Robert D. 1989. “Dimensional interactions between color and pitch.” Journal of Experimental Psychology: Human Perception and Performance 15: 69–79.

Marks, Lawrence E. 1974. “On associations of light and sound: The mediation of pitch, loudness, and brightness.” American Journal of Psychology 87: 173–188. Marks, Lawrence E. 1982. “Bright sneezes and dark coughs, loud sunlight and soft moonlight.” Journal of Experimental Psychology: Human Perception and Performance 8: 177–193. Marks, Lawrence E. 1987. “On crossmodal similarity: Auditory-visual interactions in speeded discrimination.” Journal of Experimental Psychology: Human Perception and Performance 13: 384–394. Marks, Lawrence E. 2004. “Cross-modal interactions in speeded classification.” In Handbook of Multisensory Processes, edited by Gemma Calvert, Charles Spence, and Barry E. Stein, 85–106. Cambridge, MA: MIT Press. Marks, Lawrence E. 2014a. “Synesthesia: A teeming multiplicity.” In Varieties of Anomalous Experience: Examining the Scientific Evidence (Revised Edition), edited by Etzel Cardeña, Steven J. Lynn, and Stanley C. Krippner, 79–108. Washington, DC: American Psychological Association. Marks, Lawrence E. 2014b. “Weak synesthesia in the general population.” In Oxford Handbook of Synesthesia, edited by Julia Simner and Edward M. Hubbard, 761–789. Oxford: Oxford University Press.

THEORIES ON DIGITAL SYNESTHESIA

Mroczko-Wąsowicz, Aleksandra and Danko Nikolić. 2014, “Semantic mechanisms may be responsible for developing synesthesia.” Frontiers in Human Neuroscience 10:3389. Published August 19, 2014. doi.org/10.3389/fnhum.2014.00509

Wicker, Frank W. 1969. “Mapping the intersensory regions of perceptual space.” American Journal of Psychology 81: 178–188. Wundt, Wilhelm. 1874. Grundzüge der Physiologischen Psychologie. Leipzig; Engelmann.

Lawrence E. Marks / Catherine M. Mulvenna

This article is dedicated to the memory of Frans Evers (1948–2010).

brighter in color and smaller in size? Auditory –visual correspondences in 10-month-old infants.” Infant Behavior and Development 35: 727–732.

Mulvenna, Catherine M. 2012. On creativity in synaesthetes: Roles of neural connectivity, Cognitive control, and perceptual correspondence. Doctoral thesis, University College London. Mulvenna, Catherine M. and Vincent Walsh. 2005. “Synaesthesia: supernormal integration?” Trends in Cognitive Sciences 10:350–352. Parise, Cesare V. 2016. “Crosssmodal correspondences: Standing issues and experimental guidelines.” Multisensory Research 29: in press. Spence, Charles. 2011. “Crossmodal correspondences: A tutorial review.” Attention, Perception, & Psychophysics 73: 1–25. Walker, Peter, J. Gavin Bremner, Ushi Mason, et al. 2010. “Preverbal infants’ sensitivity to synaesthetic crossmodality correspondences.” Psychological Science 21: 21–25. Walker, Peter and Sylvia Smith. (1985). “Stroop interference based on the multimodal correlates of haptic size and auditory pitch.” Perception 14: 729–736.

34

Synesthesia: Perception, Language, Digital Art

Dedication

A Philosophical Perspective on Unified Conscious Experience in Synesthesia: Insights from Philosophy of Perception and Aesthetics Aleksandra Mroczko-Wąsowicz

 Keywords   philosophy of mind, philosophy of perception, aesthetics, synesthesia, the unity of consciousness, multisensory integration, cross-modal binding, semantics  

Philosophy of perception and aesthetics are disci-

of perception as including the mentioned non-sensory

plines about experiencing the world. Philosophy of

aspects, and in consequence a liberal understanding

perception is about perceptual experiences. Aesthet-

of the philosophy of perception as a discipline going

ics is about aesthetic experiences, e.g., the experience

well beyond the restrictive characterization of per-

we have when looking at digital art works, or the

ception. In this way, philosophical questions about

experience we have when we identify with Wassily

perceptual and related non-perceptual mental

Kandinsky or any other synesthete. This paper

processes cover not only philosophy of perception

acknowledges a wide scope of perceptual experiences,

and aesthetics, but also a significant part of the phi-

i.e., not exclusively identified with the sensory stimula-

losophy of mind. Many issues in aesthetics may be

tion. Accordingly, what belongs to perception includes

considered to be topics in philosophy of perception

various cross-modal interactions, categorizing,

and philosophy of mind (Nanay 2015; Noe ¨ 2015). One

conceptualization processes, different kinds of top-

example is the question of whether aesthetic prop-

down non-sensory influences, and attention effects.

erties such as beauty, harmony, or coherence can be

This claim lays the foundation for an expansive view

perceived or only inferred from other properties. This

35

has recently become a part of the broader debate

17th and 18th centuries by philosophers John Locke

in the philosophy of perception, namely the discussion

(1689) and Gottfried Wilhelm Leibniz (1765), reporting

on whether high-level properties (i.e., properties

of a blind man for whom thinking of the color scarlet

over and above those directly transduced by the

was like hearing the sound of a trumpet. Nowadays

sensory modalities) are admissible for entry into the

we know more about this condition and its various

contents of perception and phenomenal conscious-

forms, but the phenomenon of synesthesia still poses

ness (Macpherson 2011b; Siegel 2010; Bayne 2009).

questions for philosophers, psychologists, and cogni-

These properties are meant to be abstract, generalized,

tive neuroscientists.

tioned aesthetic properties, kind properties

tions of synesthetic perception are complementary,

(recognizing that something belongs to a certain

but a clear distinction should be made between

category), numerical values, and other semantic

the involuntary psychological phenomenon of

properties representing some meaning such as a

synesthesia occurring in a minority of subjects and

number or a letter. In this paper I consider conscious

synesthesia in art involving the intentional creation of

experience in synesthetic perception to be a striking

multimedia that share simultaneously produced

example supporting the liberal thesis that contents of

features, e.g., an association between audition and

perceptual experience may include high-level prop-

vision by sharing data or computational process (Sagiv

erties; this means that synesthetes experience their

2009). Genuine synesthetes may experience color

extraordinary percepts as having high-level properties

photisms on some internal mental screen if they are

(Mroczko-Wąsowicz 2011; Mroczko-Wąsowicz and

associators. If they are projectors they may project

Nikolić 2014; Matey 2014).

concurrent colors onto the inducing stimulus, onto

We usually experience the world in a multisensory

a bodily reference frame such as their hands, or

manner. Our senses are not independent streams of

into the immediate peripersonal space within their

information. They influence each other so that respec-

reach. This is unlikely in cases of creative cross-modal

tive sensations converge into a unified experience.

interactions, except for the last option which seems

For some individuals, stimulation of one sensory or

prospective for synesthesia in digital arts.

cognitive stream may produce perceptual experience

There are numerous forms of developmental

in more than one modality. In such a “startling

synesthesia (i.e., healthy persons who have had

sensory blending” (Cytowic, 1997, p. 17) synesthetes

this condition for as long as they can remember)

have “a conscious experience of systematically

differing from each other not only in their respective

induced sensory attributes that are not experienced

inducer-concurrent pairings but also in etiology.

by most people under comparable conditions”

It is common to all these cases that the internally

(Grossenbacher and Lovelace 2001, 36). As a result,

generated synesthetic concurrent is an actual

the stimuli corresponding to the inducer and the

perceptual experience or sensory imagery, and not

experiences associated with the concurrent form an

merely a metaphor or association (Baron-Cohen

integrated percept (Mroczko-Wąsowicz and Werning

et al. 1987; Martino and Marks 2001; Macpherson

2012; Mroczko-Wąsowicz 2011).

2007; Deroy 2015). Developmental synesthesia

Although recent scientific studies examine cross-modal interactions, philosophers and artists

is inborn and consistent throughout a synesthete’s lifetime. There are, however, other kinds of

have already prepared the ground and initiated dis-

synesthetic-like experiences that may be induced

cussions on multisensory perception. Some promi-

temporarily under hypnosis, hallucinogenic drug-

nent examples include philosopher John Locke, the

influence, or acquired as more stable phenomena

mentioned synesthetic painter Wassily Kandinsky, and

after sense-organ or brain injury (Brogaard 2013;

composers Richard Wagner and Alexander Scriabin.

Luke and Terhune 2013; Deroy and Spence 2013;

The first remarks about synesthesia were made in the

Cohen Kadosh 2009).

THEORIES ON DIGITAL SYNESTHESIA

Aleksandra Mroczko-Wąsowicz

Philosophical, psychological, and artistic explora-

36

A Philosophical Perspective on Unified Conscious Experience in Synesthesia: Insights from Philosophy of Perception and Aesthetics

and cognitive in their nature. Examples are the men-

Synesthetic experiences are not just cross-modal,

used as a test case for philosophical theories of

but they also cross domains, i.e., besides the various

perception. Since synesthesia crosses modalities and

modalities of senses, these experiences may involve

domains, it is capable of challenging traditional

domains of bodily, motor and emotional states as

methods which make clear-cut distinctions individ-

well as domains of abstract, conceptually-represented

uating different sensory modalities as independent

entities like numbers or time units (Mroczko-

senses (O’Callaghan 1998; Gray 2011; Keeley 2013),

Wasowicz and Werning 2012). Cases of synesthetic

or separating various mental faculties or levels of

inducers going beyond traditionally denoted sensory

processing (Sagiv et al. 2011; Mroczko et al. 2009;

modalities have been found for activities such as read-

Mroczko-Wąsowicz and Nikolić 2013). Considering

ing musical notes, calculating, imagining, or just think-

the relationship between synesthetic and non-

ing of a stimulus (Dixon et al. 2000; Ward, Huckstep,

synesthetic perception can make us think of defini-

and Tsakanikos 2006; Simner 2007). For many forms of

tions of perceptual experience as being too narrow

synesthesia the inducers are human artifacts, various

and excessively rigid. It is not only individual

cultural constructs, e.g., letters, numbers, time units

differences that seem neglected or underestimated.

(days of the week, months in the calendar), words,

Understanding perception is often limited to sensory-

musical notes and instruments, or sports-specific

based veridical representations. However, a thorough

movements, e.g., swimming styles (Ward, Tsakanikos,

insight into perception should include a broad

and Bray 2006; Rich et al. 2005; Mann et al. 2009;

spectrum of top-down factors and their interactions

Nikolić et al. 2011). The categorization of these events/

with bottom-up features. Synesthesia, being clearly

objects is likely high-level, involving a conceptual

a cogno-sensory phenomenon, poses difficulties for

component. Recent studies suggest an important

conservative theories of perception, theories which

role of semantics in the induction of synesthesia. The

do not accept the inclusion of high-level (represen-

phenomenon seems to rely on a certain interpretation

tations of) properties in the contents of perception.

of the stimulus and the meaning that it has for the

Overall, synesthesia encourages the thought of

subject (Ward and Simner 2003; Ward and Sagiv 2007;

perception as a more diversified phenomenon than

Mroczko-Wąsowicz and Nikolić 2014; Rothen et al.

is generally assumed.

2013; Meier 2013; Chiou and Rich 2014). For example,

When we experience different sensory features at

synesthetic colors elicited by ambiguous graphemes

the same time, even though they do not belong to one

depend on the interpretation of the graphemes as

and the same object, we can still experience them to-

either a numeral or a letter (Dixon et al. 2006).

gether, i.e., as copresent or unified in our experience.

Research on synesthesia can offer profound

This is called phenomenal unity of consciousness.

implications for considerations in philosophy of

However, when the features belong to a single object,

mind and philosophy of perception, in particular on

they are experienced not only as copresent but as

such topics as theories of the unity of consciousness,

constituting one and the same object. This is known

cross-modal experiences, sensory integration, and

as object unity. We often switch between these kinds

the binding problem (Hardcastle 1998; Bayne 2010;

of unity as well as between their various degrees

Mroczko-Wąsowicz 2011; Macpherson 2007, 2011a;

of strength as a result of attention processes and

Mroczko-Wąsowicz and Werning 2012) as well as

mechanisms of multisensory integration (Deroy 2014;

debates on mental architecture, modularity of mind,

cf., Bayne 2010; Mroczko-Wąsowicz 2011). In synes-

and cognitive penetrability of perception (Fodor 1983;

thetic perception these unities take place together

Pylyshyn 1999; Baron-Cohen et al. 1993; Segal 1997;

as phenomenal object unity of consciousness. This

Macpherson 2012; Siegel 2012; Mroczko-Wąsowicz

means synesthetes are able to assign additional

and Nikolić 2014). A good theory must be able to

phenomenal properties to certain objects. The

accommodate how a system usually works and how

phenomenon of synesthesia exhibits a multitude of

it deviates from such functioning. Synesthesia can be

experiential levels driven by sensory, motor, bodily,

37

a rich diversity of phenomenal contents permanently

connectivity between distinct brain areas which

and perceptually bound together within a single uni-

facilitates mappings between concepts, experiences,

fied conscious experience (Mroczko-Wąsowicz 2011).

and behaviors (Ramachandran and Hubbard 2001).

These contents represent both high- and low-level

Synesthetic abilities are quite common among artists

properties. In this way, synesthesia transgresses the

who use creative multisensory interactions in their

boundaries between perception, action, and cogni-

works. There are various methods of communicating

tion, i.e., between capacities traditionally considered

synesthetic experiences. The preferred techniques

in philosophy, psychology, and cognitive science

deployed by synesthetic artists allow those aspects of

as independently operating domains (Mroczko-

their synesthesia that are otherwise inexpressible

Wąsowicz and Werning 2012; Mroczko-Wąsowicz

to be expressed (Steen 2001, 203). The artworks

2015). The resulting conscious experiences in

inspired by synesthesia may convey some information

synesthesia are typically synchronically unified,

about the phenomenology of this condition. As has

meaning that all the sub-experiences at a given time

already been mentioned, artistic practice and work

are present simultaneously, forming an overall

in aesthetics are relevant to debates in philosophy

phenomenal perspective — a single encompassing

of mind and philosophy of perception, especially

phenomenal state.

those concerning imagination, perception, judgment,

be co-conscious parts or aspects of the subsuming

and the construction of meaning (Noe ¨ 2015). Investigating these issues while taking synesthesia

state which has a conjoint phenomenology, or a joint

research into consideration seems likely to benefit

The different component experiences are said to

phenomenal content (Shoemaker 1996, 2003; Bayne

all the disciplines involved. This provides a sufficient

and Chalmers 2003; Tye 2003; Raymont and Brook

reason to study the phenomenon of synesthesia

2009). As such, they are intimately linked to and

from different epistemic perspectives using

integrated from the perspective of the self. The phenomenal object unity experienced at a particular point in time (e.g., when enjoying synes-

philosophy, science, and the arts. Such an interdisciplinary cooperation may well generate novel insights into human cognition.

thetic inducer-concurrent couplings) is a functional property, multi-realizable by a variety of synesthetic conditions. Also, it is not a binary feature of conscious experience, either fully present or not at all. The gradual instantiation of the unity becomes evident when comparing the experiences of two main groups of synesthetes (associators and projectors) and their different spatial frame of reference, including different perceptual salience as well as when contrasting synesthesia with synchronesthesia, i.e., common multisensory perception of non-synesthetes. This suggests that such a phenomenal coherence  / qualitative congruence comes in degrees which may correspond to various degrees of cross-modal binding. Although the links between synesthesia and creativity remain unclear, there is some evidence of a tendency for synesthetes to be more engaged in art (Ward et al. 2008; van Campen 2007; Rothen and Meier 2010). In addition, some researchers suggest

THEORIES ON DIGITAL SYNESTHESIA

38

Aleksandra Mroczko-Wąsowicz

that synesthetes and artists inclusively exhibit greater

A Philosophical Perspective on Unified Conscious Experience in Synesthesia: Insights from Philosophy of Perception and Aesthetics

emotional, and conceptual systems. Thus, it combines

Acknowledgments This work was supported by the Ministry of Science and Technology (project: MOST104-2628-H-010002-MY3).

References Baron-Cohen, S., Wyke, M., and Binnie, C. (1987). Hearing words and seeing colours: an experimental investigation of a case of synaesthesia. Perception 16, 761–767. doi:10.1068/p160761 Baron-Cohen, S., Harrison, J., Goldstein, L. H., and Wyke, M. (1993). Coloured speech perception: Is synaesthesia what happens when modularity breaks down? Perception 22, 419–426. Bayne, T. (2009). Perception and the reach of phenomenal content. Philos.Q. 59, 385–404. doi:10.1111/j.1467-9213.2009.631.x Bayne, T. (2010). The Unity of Consciousness. Oxford: Oxford University Press. Bayne, T. and Chalmers, D. (2003) What is the unity of consciousness? In A. Cleeremans (Ed.) The Unity of Consciousness: Binding, Integration and Dissociation. Oxford: Oxford University Press. Brogaard, B. (2013). Serotonergic hyperactivity as a potential factor in developmental, acquired and drug-induced synesthesia. Frontiers in Human Neuroscience 7:657. doi: 10.3389/fnhum.2013.00657 Chiou, R., and Rich, A.N. (2014). The role of conceptual knowledge in understanding synesthesia: evaluating contemporary findings from a “hub-and-spokes” perspective. Frontiers in Psychology 5:105. doi:10.3389/fpsyg.2014.00105 Cohen Kadosh, R., Henik, A., Catena, A., Walsh, V., & Fuentes, L.J. (2009). Induced Cross-Modal Synaesthetic Experience Without Abnormal Neuronal Connections. Psychological Science, 20, 258–265. Deroy, O. (2014). The Unity Assumption and the Many Unities of

Consciousness. In D.J. Bennett and C.S Hill (eds.) Sensory Integration and the Unity of Consciousness. Cambridge, MA: MIT Press, 105–124.

Macpherson, F. (2011a). CrossModal Experiences. Proceedings  of the Aristotelian Society 111 (3pt3):429–468.

Deroy, O. (2015). Can sounds be red? A new account of synaesthesia as enriched experience. In Coates, P. & Coleman, S. (eds.). Phenomenal qualities. Oxford: Oxford University Press.

Macpherson, F. (2011b) The admissible contents of experience. In: Macpherson, F. and Hawley, K. (eds.) The Admissible Contents of Experience. Wiley-Blackwell.

Deroy, O. and Spence, C. (2013). Training, hypnosis, and drugs: artificial synaesthesia, or artificial paradises?. Frontiers in Psychology 4:660. doi: 10.3389/fpsyg.2013.00660 Dixon, M. J., Smilek, D., Cudahy, C., and Merikle, P. M. (2000). Five plus two equals yellow. Nature 406, 365. Dixon, M.J., Smilek, D., Duffy, P. L., Zanna, P. M., and Merikle, P. M. (2006). The Role of Meaning in GraphemeColour Synaesthesia, Cortex 42, 243–252. Fodor, J. A. (1983). The Modularity of Mind. Cambridge, MA: MIT Press. Gray, R. (2011). On the nature of the senses. In F. Macpherson (ed.) The Senses: Classic and contemporary philosophical perspectives, (New York: OUP), 243–260. Hardcastle, V. (1998). The binding problem. In W. Bechtel and G. Graham (eds.) A Companion to Cognitive Science, Oxford: Blackwell Publishers. Keeley, B. L. (2013). What exactly is a sense? In J. Simner and E. Hubbard (eds.) The Oxford Handbook of Synaesthesia, (Oxford: OUP), 941–958. Luke, D.P. and Terhune, D.B. (2013). The induction of synaesthesia with chemical agents: a systematic review. Frontiers in Psychology 4:753. doi: 10.3389/fpsyg.2013.00753 Locke, J. (1689). An Essay Concerning Human Understanding, Clarendon Edition, ed P.H. Nidditch (Oxford: Clarendon Press); (repr. 1979). Macpherson, F. (2007). Synaesthesia, Functionalism and Phenomenology.  In: De Caro, M., Ferretti, F. and Marraffa, M. (eds.) Cartographies of the Mind. Philsophy and Psychology in Intersection. Series: Studies in brain and mind, 4. Springer Verlag.

Macpherson, F. (2012). Cognitive penetration of colour experience: rethinking the issue in light of an indirect mechanism. Philos Phenomen Res 84, 24–62. Mann, H., Korzenko, J., Carriere, J.S.A., and Dixon, M.J. (2009).Timespace synaesthesia – a cognitive advantage? Consciousness and Cognition 18, 619–627.doi:10. 1016/j. concog.2009.06.005 Martino, G., and Marks, L.E. (2001). Synesthesia: strong and weak. Curr. Dir. Psychol. Sci. 10, 61–65 10.1111/1467-8721.00116 Matey, J. (2014). Can Blue Mean Four? In D.J. Bennett and C.S Hill (eds.) Sensory Integration and the Unity of Consciousness. Cambridge, MA: MIT Press, 151–170. Meier, B. (2013). Semantic representation of synaesthesia. Theoria et Historia Scientiarum 10, 125–134. http:// dx.doi.org/10.12775/ths-2013-0006 Mroczko-Wąsowicz, A. (2011 /2013). The Unity of Consciousness and the Phenomenon of Synesthesia [Die Einheit des Bewusstseins und das Phänomen der Synästhesie]. Published doctoral dissertation, Johannes Gutenberg University of Mainz.

Mroczko, A., Metzinger, T., Singer, W., and Nikolić, D. (2009). Immediate transfer of synesthesia to a novel inducer. Journal of Vision 9, 25, 1–8. doi: 10.1167/9.12.25 Mroczko-Wąsowicz, A., and Nikolić, D. (2013). Colored alphabets in bilingual synesthetes. In: J. Simner and E. Hubbard (Eds.) Oxford Handbook of Synesthesia, Oxford: Oxford University Press, 165–180. Mroczko-Wąsowicz, A., and Nikolić, D. (2014). Semantic mechanisms may be responsible for developing synesthesia. Frontiers in Human Neuroscience 8:509. doi: 10.3389/fnhum.2014.00509 Mroczko-Wąsowicz, A., and Werning, M. (2012). Synesthesia, sensory-motor contingency and semantic emulation: how swimming style-color synesthesia challenges the traditional view of synesthesia. Frontiers in Psychology 3:279. doi:10.3389/fpsyg.2012.00279 Nanay, B. (2015). Using Philosophy of Perception in Aesthetics. Aesthetic Investigations, 1 (1), 174–180. Nikolić, D., Jürgens, U.M., Rothen, N., Meier, B., and Mroczko A. (2011). Swimming-style synesthesia. Cortex 47, 874–879. Noe¨ , A. (2015). Strange Tools: Art. And Human Nature, New York: Farrar, Straus and Giroux / Hill and Wang. O’Callaghan, C. (1998). Seeing what you hear: Cross-modal illusions and perception. Philosophical Issues 18, 316–338. Pylyshyn, Z.W. (1999). Is vision continuous with cognition? The case for cognitive impenetrability of visual perception. Behav. Brain Sci. 22, 341–365. doi: 10. 1017/ s0140525x99002022

Mroczko-Wąsowicz, A. (2015). What Can Sensorimotor Enactivism Learn from Studies on Phenomenal Adaptation in Atypical Perceptual Conditions? – A Commentary on Rick Grush and Colleagues. In T. Metzinger & J. M. Windt (Eds). Open MIND: Philosophy and the Mind Sciences in the 21th Century. Cambridge, MA: MIT Press, 16(C), 1-17. doi: 10.15502/9783958570603

Ramachandran V.S., Hubbard E.M. (2001). Synaesthesia: A window into perception, thought and language. Journal of Consciousness Studies, 8 (12):3–34.

Mroczko-Wąsowicz, A. (forthcoming). Editorial: Perception-Cognition Interface & Cross-Modal Experiences. Insights into Unified Consciousness. Frontiers in Psychology/ Specialty: Consciousness Research.

Raymont, P., Brook, A. (2009). The Unity of Consciousness. In: B.P. McLaughlin, A. Beckermann, S. Walter (Eds.) The Oxford Handbook of Philosophy of Mind. Oxford: Clarendon Press.

39

Rothen, N., and Meier, B. (2010). Higher prevalence of synaesthesia in art students. Perception ,39(5), 718–720. doi:10.1068/p6680  Rothen, N., Nikolić, D., Jürgens, U.M., Mroczko-Wąsowicz, A., Cock, J., and Meier, B. (2013). Psychophysiological evidence for the genuineness of swimming-style colour synaesthesia. Conscious Cogn 22, 35–46. Sagiv, N.,  Dean, R.T., and Bailes, F. (2009). Algorithmic Synaesthesia. In: R.T. Dean (Ed.), The Oxford Handbook of Computer Music, New York: Oxford University Press, 294–311.

Segal, G. (1997). Synaesthesia: Implications for the modularity of mind. In S. Baron-Cohen and J.E. Harrison (ed.) Synaesthesia: Classic and Contemporary Readings (Blackwell, Oxford), 211–224. Shoemaker, S. (1996). Unity of consciousness and consciousness of unity. In The First-Person Perspective and Other Essays. Cambridge: Cambridge University Press Shoemaker, S. (2003). Conscious ness and Co-Consciousness. In: A. Cleeremans (Ed.) The Unity of Consciousness: Binding, Integration and Dissociation. Oxford: Oxford University Press, 59–71. Siegel, S. (2010). The Contents of Visual Experience. New York: Oxford University Press.

Siegel, S. (2012). Cognitive Penetrability and Perceptual Justification. Noûs, 46, 201–222. Simner, J. (2007). Beyond perception: synaesthesia as a psycholinguistic phenomenon. Trends Cogn. Sci. 11, 23–29. Steen, Carol (2001). Visions Shared: A Firsthand Look into Synesthesia and Art. Leonardo, 34 (3), 203-208. Tye, M. (2003). Consciousness and Persons: Unity and Identity. Cambridge, MA: MIT Press. van Campen, C. (2007). The Hidden Sense. Synesthesia in Art and Science. Cambridge: MIT Press. Ward, J., Huckstep, B., Tsakanikos, E. (2006). Sound-colour synaesthesia: To what extent does it use cross-modal mechanisms common to us all? Cortex, 42: 264–280.

Ward, J., and Sagiv, N. (2007). Synaesthesia for finger counting and dice patterns: A case of higher synaesthesia? Neurocase 13, 86–93. Ward, J., and Simner J. (2003). Lexical-gustatory synaesthesia: Linguistic and conceptual factors. Cognition, 89: 237–261. Ward, J., Thompson-Lake, D., Ely, R., and Kalinski, F. (2008). Synaesthesia, creativity and art: What is the link? British Journal of Psychology, 99, 127–14. Ward, J., Tsakanikos, E., and Bray, A. (2006). Synaesthesia for reading and playing musical notes. Neurocase 12, 27–34. doi:10.1080/13554790500 473672

A Philosophical Perspective on Unified Conscious Experience in Synesthesia: Insights from Philosophy of Perception and Aesthetics

Sagiv N., Ilbeigi A., Ben-Tal O. (2011). Reflections on synaesthesia,

perception and cognition. Intellectica 55, 81–94.

Aleksandra Mroczko-Wąsowicz

Rich, A.N., Bradshaw, J.L., and Mattingley, J. B. (2005). A systematic, large-scale study of synesthesia: Implications for the role of early experience in lexicalcolor associations. Cognition, 98, 53–84.

THEORIES ON DIGITAL SYNESTHESIA

40

Ideasthesia and Art

Danko Nikolić

  Keywords   ideasthesia, art, synesthesia, aesthetics, beauty, perception, colors, entertainment, science, balance

Abstract Ideasthesia can be defined as a phenomenon in which

underlie the creation and consumption of art. Here, an

activation of concepts produces phenomenal experi-

attempt is made to explain the psychology of art, as

ence. The present article is concerned with the

well as define art, based on the theory of ideasthesia.

relationship between ideasthesia and art. In the past,

According to the present theory, art happens when

it has proven difficult to come up with a comprehen-

the intensities of the meaning produced by a certain

sive definition of art. Equally difficult seems to be to

creation and the intensities of the experiences induced

understand which psychological processes specifically

by that creation, are balanced out.

41

The word ideasthesia comes from Ancient Greek

get the color of the Latin A. Much other evidence also

words idea (for concept) and aesthesis (for sensation).

exists (Simner et al. 2006, Novich et al. 2011, Chiou

Hence, the term ideasthesia means sensing concepts.

and Rich 2014, Sagiv et al. 2006, Ward and Sagiv 2007,

The theory of ideasthesia was initially developed on

Dixon et al. 2000, Nikolić et al. 2011). These results could not be explained by direct

people who have additional sensory-like experiences

connections between brain areas. Ideasthesia needed

to a stimulus that otherwise would not induce such

to be introduced. The theory of ideasthesia states that

experiences. For example, every letter of the alphabet

concepts precede sensory-like experiences in synes-

may induce the experience of a certain color (letter A

thesia (Nikolić 2009). Thus, it is first the concept of the

often happens to be red). And there are many other

letter A that is activated and then, as a consequence

forms of synesthesia (Cytowic and Eagleman 2009).

of the activation of that concept, a synesthetic color

Initially, it was believed that synesthetic experienc-

“lights up” in the minds of synesthetes. The theory

es occur as a result of direct connections between

of ideasthesia emphasizes a tight relationship between

corresponding parts of the brain. For example, the

concepts; i.e., our understanding of the stimuli and

brain area responsible for grapheme detection would

the experiences i.e., the way we feel about that

somehow (aberrantly) be connected to the brain area

world (Mroczko-Wasowicz and Nikolić 2014; van

responsible for color perception. However, this hy-

Leuween et al. 2015).

pothesis had to be abandoned over time as more and

Importantly, a concept is something fundamental-

more evidence suggested that synesthesia was not

ly different than a mere direct connection between

based on direct associations, but instead that con-

brain areas. Concepts are related to our capability of

cepts mediate those associations.

engaging our full intellect to understanding the world

For example, experiments showed that synes-

around us, and of acquiring novel insights about that

thetes would associate different synesthetic colors to

world (e.g., Fodor 1998). There is no consensus on

one and the same stimulus, depending on how they

how the brain implements concepts. One recent theo-

interpreted that stimulus (Dixon et al. 2006). It has also

ry proposes that we apply concepts whenever nerve

been shown that new synesthetic associations could

cells adapt to external stimuli; and thus, that we learn

be created within minutes — simply by giving new

new concepts as our neurons learn how to perform

meaning to a certain stimulus (Mroczko et al. 2009).

such adaptations. This theory, named practopoiesis,

For example, when synesthetes were exposed to an

presumes that the mind results from a hierarchy of

unfamiliar writing system they would associate colors

adaptive mechanisms (Nikolić 2015). According to

to new letters as soon as they learned which of the

practopoiesis, the “idea” component of ideasthesia

Latin letters they corresponded to; a new “A” would

involves fast processes of neural adaptation.

Danko Nikolić

the basis of research in synesthesia. Synesthetes are

Since its introduction, the theory of ideasthesia has

consistent associations. They discovered that subjects

been extended to everyday perceptual processes.

overwhelmingly agreed about many other properties

Milan et al. (2014) investigated what kind of personality

assigned to the shapes. For example, Kiki was nervous

traits people tend to assign to two different shapes,

and high-class. Bouba was lazy and easy going. They

known as Kiki and Bouba (Fig. 1; Köhler, 1947).

concluded that sensations produced by a “spiky shape”,

Before their research, it was well known that people

a “nervous personality”, “laziness”, much like those of

consistently assign the name Kiki to the spiky star and

the sounds making the name “Kiki”, are all connected

Bouba to the round blob. However, what Milan and

in an associative network. These types of associations

colleagues found was a much more extensive form of

are also known as cross-modal associations (e.g.,

THEORIES ON DIGITAL SYNESTHESIA

42

Ideasthesia and Art

Ideasthesia in Everyday Perception

Spence 2011); much like the color blue is perceived as

between semantics and sensations in such a way that

cold and red as warm. Importantly, as

the theory of ideasthesia would apply. This was the

Milan and colleagues noted, this network of

first proposal for ideasthesia being used outside of the

cross-modal associations had stark similarities to

realm of synesthesia and being generalized to every-

the semantic associative network (e.g., “doctor” is

day perceptions. Since then, further proposals have

associated with “nurse”). But these were associations

been made on how ideasthesia may help us better

among sensations, not among concepts. Therefore,

understand consciousness (Mroczko-Wasowicz and

they concluded that there is a strong similarity

Nikolić 2014; van Leuween et al. 2015).

Ideasthesia and Art Perhaps the most surprising consequence of intro-

offer a certain intellectual value? For example, what

ducing the concept of ideasthesia has been in its

is aesthetically different about the way Dostoyevsky

relationship to art. I have been surprised to witness

depicts Russian society in the 18/19th century and how

many artists adopting the concept in describing or

historians do so?

even naming their pieces of work. I have found these

Many attempts have been made at defining art,

concepts being used in a wide range of art forms

though this endeavor has proved difficult (Painter

including: paintings, music, photography, interactive

2002). The challenge is to make a distinction precisely

art, acting, installations, and even perfumes. When

between where the art is and where it ends. A

talking to some of those artists, I learned that they

successful definition would minimize false inclusions

overwhelmingly felt that ideasthesia somehow

and false exclusions, i.e., the number of works that

described the very process by which they created art.

would be accepted as art but would not be covered by

This adoption of ideasthesia by artists also pro-

the definition and vice versa, those that the definition

vokes a question: Can we use ideasthesia to formulate

would cover but that would generally be rejected.

a theory of art? Over time, I have come to believe that

A successful definition would also account for the

the answer to this question is affirmative. The relation-

subjectivity of art.

ship between the two opposing forces of ideasthesia,

The definition that follows also offers a theory

i.e., the concept and the sensation, can be used to for-

of art and is not made by an art historian, but by a

mulate a hypothesis about psychological events that

cognitive scientist. It may well be that the art prob-

underlie the process of either creating an art piece

lems presented here are incomplete and somewhat

or appreciating (i.e., consuming) it. I call the theory:

naïve. In any case, an effort to bridge these distant

ideasthesia balance theory.

fields — cognitive science and art theory — has

The key problem here is to uniquely distinguish art from all other, somewhat related human activities.

been made. The present theory is a psychological theory; it

For example, we can ask the question: What precisely

postulates what is happening within the minds of

distinguishes one novel that is generally judged as a

people when they experience art. Here, we try to

valuable piece of art from another novel that clearly

understand what the psychological processes are that

does not get such appreciation? Or, what is the

lead a person to declare some act or artifact as a

fundamental aesthetic difference between a “soap”

piece of art. The theory should apply to both the

opera as compared to a Shakespeare play? Likewise,

actual act of creating art and the consumption of art.

one can ask: What is it that distinguishes insightful

At the heart of the present theory lies the relation-

intellectual works that are unanimously considered

ship between meaning and experience as the two

not to be art, such as e.g., a doctoral thesis, from

forces of ideasthesia. By meaning, what I refer to here

other works that are considered a piece of art but also

far exceeds the type of meaning that can be expressed

43

matter what the insight is about. It can be about

linguistics. Here, under “extraction of meaning” from

academic content such as a mathematical theorem,

a stimulus we assume all of the changes made by our

but also about social life or even our basic perceptions

nervous system to prepare us for interaction with

such as colors, sounds, movements, or space. Whatever

that stimulus (Nikolić 2015); irrespective of whether

there is that we can process, memorize and under-

these changes can be described by words or not. In

stand, this same thing can also be probed by stimuli

ideasthesia, meaning is much wider than what can

that are either more or less meaningful. Sensation, on the other hand, as the second part of ideasthesia, is related to phenomenal experiences,

is largely involved in the extraction of meaning (e.g.,

also known as qualia. Sensations are about the way

Kuipers et al. 2013). Similarly, the number of con-

things feel and “are like”. It is about the redness of a

cepts that we have is far larger than can be named

red color, and the sourness of a lime. Sensations make

or expressed in words. Therefore, the meanings that

up our inner mental life and “light it up” so that life

relate to language are only a small fraction of the total

does not happen “in the dark”.

meaning machinery that our minds are equipped with. Consequently, by meaning I refer here to the full

To formulate the ideasthesia balance theory it is necessary first to establish that both meanings and

depth of relationship an art piece has to a person’s

sensations can vary in magnitude. Stimuli may be

knowledge. Every individual has a certain amount

more or less meaningful, and may have more or

of life experience, defining that person’s knowledge

less intensive sensations. For example, a light touch

about the life and world. This knowledge can be

to someone’s skin may not be noticed, and hence

described as an associative network of concepts, or

a sensation might be considered weak. In contrast,

alternatively as a hierarchy of concepts. The contents

a more forceful touch may produce a stronger

of knowledge include factual knowledge, episodic

sensation. Likewise, the same light touch may occur

memories of events that occurred in a person’s life,

accidentally in a crowded street and be thus meaning-

habits, general principles of understanding how the

less, or might indicate an important sign of affection

world works, our skills, personal values, learned

and hence carry a lot of meaning.

emotional responses, fears, hopes, individual philosophy of life, and so on. For the most part

In our mental life, meaning and sensation interact, and their relationship can be rich. We have discussed

this knowledge is unconscious, and hence not easy

cases above in which meaning induces sensation. We

to describe with words.

have also seen similarities between the network of the

These components of our knowledge do not stand

qualities of sensations and the network of concepts

independently, but are related and connected. One

underlying the extraction of meaning. But there are

meaningful aspect of any stimulus, be it art or not, is

more relationships between the two. Firstly, extracted

that which taps into the vast knowledge that a person

meaning can alter the perceived strength of sensation;

comes with and uses it to interact with the world. A

everything else being equal, a meaningful stimulus

stimulus that is particularly strong on meaning is one

is likely to induce a stronger sensation than a less

that probes our existing knowledge at a deeper level;

meaningful one. Also, the intensity of a sensation can

a meaningful stimulus powerfully affects the semantic

affect the process of learning, and can thus determine

structure that we already come with. Such a stimulus

the efficiency with which we acquire new knowledge

not only creates new memories, but makes us re-

(including new concepts).

organize the existing ones. It makes us form new con-

Ideasthesia balance theory states a particular

nections – new insights. A truly meaningful stimulus

relationship between the depth of meaning and the

makes us see some aspect of the world more clearly;

intensity of sensation: A creation that we are likely to

better, in a new light, in a more comprehensive way.

judge as art is one in which the meaningfulness and

To be declared as meaningful, it does not really

THEORIES ON DIGITAL SYNESTHESIA

the strength of sensation are well-correlated. In an art

44

Danko Nikolić

be verbalized. Hence, the theory takes into account the findings of neuroscience that the motor cortex

Ideasthesia and Art

merely by language — i.e., that which is understood by

piece, the moments (the components) that carry the

extreme point we can put entertainment, with

most meaning are also those that induce the strongest

excessive sensations and little meaning, and at the

sensations. If there is an event in a story, a sequence

other extreme point we can put various forms of

of notes in a melody, or a form in a sculpture that

scientific and technical work that are often loaded

induces the strongest sensations, this same event,

with meaning but are lacking in sensations (Fig. 4).

musical sequence and form must also be the one

Importantly, art falls exactly in the middle of that

that carries the most meaning. These moments and

continuum.

components have to be the places of insight that are perfectly combined with simultaneously inducing e.g.,

Comparison of art to entertainment: Entertainment

the strongest emotions. Therefore, art needs to effec-

runs on emotions, with an emphasis on emotions of

tively combine an insight with feeling.

positive valence. Negative emotions typically present

It is not that all human activities produce such a

an interlude to a resolution by positive ones (a happy

correlated induction of sensations and conceptualiza-

ending). Fear is followed by security; tension followed

tions. In fact, a perfect balance is difficult to achieve

by relaxation, and so on. In entertainment, this emo-

and requires hard work. It is comparably easier to

tional rollercoaster usually does not carry significant

induce strong sensations that do not have a strong

meaning. The consumer does not learn much. There

meaning attached. Also, it is comparatively easier

is no significant novel relation to the real world. In

to express meaning in a way that is quite dull with

entertainment, it does not matter whether the events

respect to the sensations that it evokes.

that unfold are realistic to life, whether the decisions

The relationships between meaning and sensa-

of characters are illogical, of whether the incidents

tions with respect to the balance of ideasthesia are

building the plot are physically impossible. The con-

illustrated in Fig. 2 – 5. Fig. 2 shows the entire

sumer of entertainment is there for the sensational

space of possible relationships between meaning

ride, not for insights and logical consistency. In con-

and sensations. However, there is only a narrow band

trast to entertainment, the sensations of an art piece

of relationships that can be considered as well

are combined with insights. The story in a drama may

balanced. Note that the absolute value of intensity is

help us learn something about real life. It may help us

not considered important; it is rather the balance of

understand ourselves or our friends. Often, art helps

the respective intensities that matters. Every art piece

make sense of emotions. For example, both James

is comprised of multiple parts and each part will carry

Bond and The Godfather movies are fictions that por-

more or less meaning or induce more or less sensation.

tray plenty of murders, explosions, fistfights, revenges,

These intensities need to be aligned as much as pos-

and erotic moments. However, only The Godfather

sible such that their idea vs. aesthesia relations lie all

tells a story of something that is realistic and from

along the gray diagonal in Fig. 2.

which one can learn about the real world: the fight for

A one-dimensional continuum of ideasthesia balance can be defined along the diagonal orthogonal to that of maximized balance (Fig. 3). This representa-

power, corruption, family relations, real conflicts in which individuals find themselves, and so on. For these reasons, art does not need to rely much

tion can indicate the overall idea vs. aesthesia balance

on happy endings. Art can also induce unpleasant

in a certain creation. The continuum

emotions, but if these emotions make sense in the

ranges from an extreme dominance of sensations

great scheme of things, i.e., if the events are mean-

(very little meaning) to an extreme dominance of

ingful overall, the consumer can take the insights that

meaning (scarce sensations).

come with the “ugly” emotions as a valuable outcome.

Such a one-dimensional representation can be

How can one handle the misfortune of the main

used to sort out various types of human creations

protagonist in Kafka’s Metamorphosis? This can

with respect to the dominance of either sensations

be done only by finding meaning in the story – for

or meaning, or their balance. For example, at one

example, by understanding it as a depiction of similar

45

Ideasthesia and Art

Danko Nikolić

Fig. 1

Fig. 2

THEORIES ON DIGITAL SYNESTHESIA

46

Fig. 1 Which one is Kiki and which one is Bouba? People overwhelmingly agree that the spiky star is Kiki.

Fig. 2 The space of possible relations between meaning and sensations. The contents of an art piece need to be aligned along the gray diagonal. Fig. 3 Definition of a one-dimensional continuum of ideasthesia balance.

Fig. 4 Entertainment, art, and science occupy different positions on the continuum of ideasthesia balance.

Fig. 3

Fig. 4

47

unsuccessful struggles happening to people in real

or anything else. However, the biggest difference from

life who may be suffering from a different nasty

art is not primarily that science is less capable of

misfortune of life than turning into a giant insect; such

generating such sensations (it is indeed less capable,

as for example a terminal disease. The beauty in this

but this is not the main difference) but that science

otherwise “ugly” story is in the balance between

cannot offer as good a balance between idea and

emotions and meaning.

aesthesia as art does. Scientific work cannot be made

Because art is meaningful in addition to often

such that it resides in the gray diagonal of Fig. 2.

giving us a good time, it provides us with a certain

The story of a scientific result is not told through a

value that exceeds that of entertainment — a value

set of rich sensory inputs. Instead, abstract (i.e.,

with a longer-lasting effect. Art builds us up, elevates

“boring”) language has to be used. In science, the point

us, and creates new experiences integrated into the

of highest meaning is not the point of strongest

semantic structures. It also expands these semantic

sensations induced. It is not that an emotional

structures. The new knowledge it imparts is likely to

equivalent of someone getting into a life-threatening

be useful in our lives at some later time; it is likely to

situation occurs at the moment of deriving the key

apply to events that actually occur to us. All else being

logical conclusion of an argument, or at the moment

equal, the person who has consumed art is likely to

of finding out that a statistical test was significant.

carry more wisdom than the one who has limited his or her experiences to mere entertainment.

Ideasthesia balance theory also tells us why art cannot be created by a forced combination of entertainment and scholarly work. One cannot create art by having characters in a soap opera spelling out

and make insights is not only a characteristic of art.

scientific facts. This would be unbalanced, and

Science and related disciplines create new knowledge

hence the resulting creation would be positioned all

as well but in a much more explicit way. Additionally,

over the graph in Fig. 2, not in the gray diagonal.

knowledge accessed through science is often more

For the same reason, one cannot simply tell jokes

direct, presented in a succinct form and is more

during a scientific presentation to turn science into art. Robert M. Pirsig (1999) argues in his book Zen and

accumulation of scientific knowledge tend to grow

the art of motorcycle maintenance that quality results

disconnected from everyday life with their specialized

from a combination of two factors: rational and

terminology, abstract concepts, and “dry” data. As a

romantic. I postulate that he is arguing something

result, they tend to produce less intensive sensations

similar to the balance of ideasthesia. The romantic

than art works. And when they do create sensations,

component relates to being “in the moment” and

these are not well-balanced with the implications of

defies rational analysis. In contrast, the rational

the scientific work. A discovery of a new molecule that

component is about details and understanding the

cures a certain disease is certainly very meaningful,

inner mechanics. Pirsig concludes that quality comes

but the way scientists describe such a molecule is by

when the two aspects are combined. In relation to

no means different from the description of any other,

ideasthesia, the romantic component may correspond

much less meaningful, molecule — it is just a formula.

to aesthesis, while the rational analysis may be the

In fact, the sensory experience is broadly similar for

idea part of ideasthesia. Therefore, to paraphrase

any formula describing any kind of molecule — and

Pirsig, high quality may be achieved when both

these experiences are not very rich at all. It is not that scientists don’t care about inducing

sensations and concepts are strong; that is, when ideasthesia is maximized.

sensation. They gladly hang onto any sensation-based aesthetics that can be associated with their abstract scientific work, be it simple symmetry, a nice fractal structure of data, an artificially-colored visualization

THEORIES ON DIGITAL SYNESTHESIA

48

Danko Nikolić

objective. However, scholarly works serving the

Ideasthesia and Art

Comparison to science: The ability to extract meaning

Implications The present theory can be used to offer an explana-

our definition of what art is. He offered an idea that

tion for the properties of art.

was revolutionary at the time.

Abstract art. A naïve view would be that abstract art

topic - that would be science. Instead, he painted a

does not carry meaning, and thus that it cannot fulfill

“black square” and he let people deal with the fear and

the criterion for the balance of ideasthesia. Where is

the insight. It is also possible that he could not have

the meaning in a bunch of circles and triangles on a

known for sure how his experiment would work out

Malevich did not write an academic essay on this

painting? Or, where is the meaning in one sequence

in the end, and may have had to deal with the same

of musical notes? The fact is that our brain extracts

fears himself. This combination of courage and insight

meaning at a level that cannot be verbalized. Meaning

makes this abstract painting a valuable piece of art.

does not require the ability to consciously express the contents. A stimulus is meaningful if it makes our

Subjectivity of art. A judgment of whether something

brain work, if it attracts our attention, and if it forces

is or is not art is highly subjective. For any particular

us to process it in some way. An input is more mean-

piece, there is no full consensus among experts. A

ingful if it requires more effort in adjusting to it. A lack

strong subjective component exists when valuing art.

of meaning comes from stimuli that we ignore or do

This is something that is perfectly expected according

not notice. Also, when we actively suppress input e.g.,

to the ideasthesia balance theory. The theory tells

filtering out one disturbing conversation in an effort

us that the artistic value of a creation can only be

to hear another, we can speak of little or no meaning

judged by the ideasthesia balance evoked within

being extracted from the suppressed stimulus.

an individual. Both the sensational and the meaning

Hence, abstract art can induce much meaning

aspects depend on the knowledge of a given

processed at a level difficult to express verbally.

person, and each individual is different. Hence, each

Also, abstract art can induce sensations. When these

individual will respond somewhat differently to

are combined well, one can achieve a balance of

any stimulus. A given creation may induce a strong

ideasthesia.

ideasthesia balance in some people and in others

However, the most powerful abstract art may be

it may not. In ideasthesia balance, art is exclusively

that which induces meaning and sensations that

defined by how a person reacts to it. This definition

transcend the modality in which the art has been

itself does not allow for any other definition of art

created. When Malevich painted his “black square”

that would be objective, i.e., that would work without

he did not primarily produce sensory sensations by

having to consider how people respond to it.


colors. Uniform black paint probably produces as little of such sensations as possible. The true sensations

Timelessness of art. Art pieces do not seem to

that he produced with that painting had to do with the

lose their effect over time. Classic paintings, classic

social aspects. Every other artist at that time would

music, classic texts; all of them continue to hold value

possibly fear stating that a black square is his own

as time goes by. In contrast, a piece of entertainment

artistic expression. One could easily be ridiculed. In

can explode in popularity for a very short period of

addition, every consumer of art or gallery custodian

time, and then it dies — it is forgotten as quickly as it

would have to deal with similar fears. And those

became popular. Moreover, the pieces that become

sensations were strong. At the same time, he offered

quickly popular end up not only less attractive soon

meaning; although not a verbal one. There was

afterward, but often even quite repulsive or annoying.

nothing much to be verbalized about a black square.

If a hit-driven radio station were to always play just

It was instead from his actions through which he

last year’s (last month’s) hits, it would quickly run

offered the idea that we can considerably broaden

out of listeners. In contrast, a classical radio station

49

can play music 100s of years old and still keep its

desirable as it enables exploration of the art piece — 

audience happy. Who wants to watch a single episode

every fresh encounter leading to something new being

of a soap opera multiple times? And yet, owning a

discovered. In this way, an art piece educates.

copy of Beethoven’s Fifth and listening to it repeatedly

It is exactly this property of consistent balance

continues to evoke pleasure. How is that possible?

between sensation and meaning — i.e., balanced

How can ideasthesia balance explain the timelessness

ideasthesia — which makes us willing to preserve art

of art and the short-lived pleasures of entertainment?

pieces, to be surrounded by them, to protect them

We have to begin by noting that our brain changes with every single exposure to a stimulus. A stimulus

from being lost, and to encourage others to become exposed to them. Also those creations do not come

is always perceived and processed a bit differently

easily. They require hard work. But once it is done

the second time around. And the changes are due

right, they provide an incalculable value for many of us.

to learning about the stimulus. Our brain learns to predict what will come next, and habituates to familiar inputs (Johnston et al. 1990, Johnston and Schwarting 1997). As a result, there is less and less need to process the stimulus with every new exposure. The novelty is reduced. The intensity of sensations is reduced. The reason that art does not annoy us with repetition and entertainment does (or art annoys us much less) lies again in the ideasthesia balance. We have to ask: How would a given creation be represented in Fig. 2? An entertaining piece is, on the whole, located in the green area and therefore it is not well balanced. However, what makes entertainment nevertheless temporarily attractive is that, when Danko Nikolić

broken down into its components, it will have at least one component that touches the gray area - something like the red line shown in Fig. 5. It is likely that the section of a piece entering the gray zone of balance first will be the most salient one — and this may make the piece an instant hit. This most salient part will be the one that will wear off most quickly after repetition, however. Then the saliency will move on to other parts, and these other parts lie outside the gray region and hence lack beauty. Suddenly, the piece turns “ugly” and unpleasant.

Ideasthesia and Art

In contrast, an art piece is much more balanced across all of its components (something like the yellow line in Fig. 5). For that reason art continues to provide pleasure even after learning. As one component sinks into the background of habituation another component surfaces, but also offers balance, and hence value and beauty. Repeated exposure is then

THEORIES ON DIGITAL SYNESTHESIA

50

Fig. 5

Fig. 5 Sensation vs. meaning balance shown for hypothetical pieces of entertainment (red), art (yellow) and science (white), as they are broken down into their components. Only an art piece is consistently located in the gray area.

51

References Chiou R, Rich AN (2014) The role of conceptual knowledge in understanding synaesthesia: Evaluating contemporary findings from a “hub-and-spokes” perspective. Fron Psychol 5:105. Cytowic RE, Eagleman D. (2009) Wednesday is indigo blue: Discovering the brain of synesthesia. MIT Press. Dixon MJ, Smilek D, Cudahy C, Merikle PM. (2000) Five plus two equals yellow. Nature. Jul 27; 406(6794):365–. Dixon MJ, Smilek D, Merikle PM (2004) Not all synaesthetes are created equal: Projector versus associator synaesthetes. Cogn Affect Behav Neurosci 4:335–343.

DeWitt MJ. (1990) Attention capture by novel stimuli. Journal of Experimental Psychology: General. Dec;119(4):397. Johnston WA, Schwarting IS. (1997) Novel popout: An enigma for conventional theories of attention. Journal of Experimental Psychology: Human Perception and Performance. Jun;23(3):622.

Nikolić D (2009) Is synaesthesia actually ideaestesia? An inquiry into the nature of the phenomenon. In: Proceedings of the Third International Congress on Synaesthesia, Science & Art, pp 26–29.

Simner J, Mulvenna C, Sagiv N, Tsakanikos E, Witherby SA, Fraser C, Scott K, Ward J (2006) Synaesthesia: the prevalence of atypical cross-modal experiences. Perception 35:1024–1033.

Nikolić D, Jürgens UM, Rothen N, Meier B, Mroczko A (2011) Swimming-style synesthesia. Cortex 47:874–879.

Spence C (2011) Crossmodal correspondences: A tutorial review. Atten Percept Psychophys 73:971–995.

Köhler W (1947) Gestalt psychology: An introduction to new concepts in modern psychology. New York: Liveright.

Nikolić D. (2015) Practopoiesis: Or how life fosters a mind. Journal of theoretical biology. May 21; 373:40–61.

Kuipers JR, van Koningsbruggen M, Thierry G (2013) Semantic priming in the motor cortex: evidence from combined repetitive transcranial magnetic stimulation and eventrelated potential. Neuroreport, 24(12), 646–651.

Novich S, Cheng S, Eagleman DM (2011) Is synaesthesia one condition or many? A large-scale analysis reveals subgroups. Journal of Neuropsychology 5:353–371.

Mroczko-Wąsowicz A, Nikolić D (2014) Semantic mechanisms may be responsible for developing synesthesia. Front Hum Neurosci 8:509.

Fodor JA. (1998) Concepts: Where cognitive science went wrong. Clarendon Press. Johnston WA, Hawley KJ, Plewe SH, Elliott JM,

Mroczko A, Metzinger T, Singer W, Nikolić D (2009) Immediate transfer of synesthesia to a novel inducer. J Vision 9.

Ward J, Sagiv N (2007) Synaesthesia for finger counting and dice patterns: a case of higher synaesthesia? Neurocase 13:86–93.

Painter C. (2002) Contemporary art and the home. Berg Publishers; Oct 1. Pirsig RM. (1999) Zen and the art of motorcycle maintenance: An inquiry into values. Random House. Sagiv N, Heer J, Robertson L (2006) Does binding of synesthetic color to the evoking grapheme require attention? Cortex 42:232–242.

Ideasthesia and Art

Danko Nikolić

Dixon MJ, Smilek D, Duffy PL, Zanna MP, Merikle PM (2006) The role of meaning in grapheme-colour synaesthesia. Cortex 42:243–252.

van Leeuwen TM, Singer W, Nikolić D. (2015) The merit of synesthesia for consciousness research. Frontiers in psychology 6.

THEORIES ON DIGITAL SYNESTHESIA

52

Transformation, Transliteration and Translation. Synesthesia and Multisensory Perception in Contemporary Visual Art Regine Rapp

  Keywords   transformation, transliteration, translation, transmodality, perception, process, algorithm, grapheme, code, monitor

The term “synesthesia”, coming from the Greek

rich 2013). Sina Trautmann-Lengsfeld claims that “the

“aisthesis” (“sensation”, “sensory impression”) and “syn”

stimulation of one sensory modality leads to [another]

(“together”), means the experience of two or more

additional, atypical sensory experience in the same or

sensory impressions at the same time. Synesthesia was

different modality” (Trautmann-Lengsfeld 2013).

not only prominent in the melding of the arts from

In my paper I want to focus on three remarkable

the Renaissance, Romanticism, Symbolism and 20th

artistic positions that approach digital synesthetic

century avant-garde; it is also a highly valuable param-

forms of transmodality in unique ways. I will intro-

eter in the field of contemporary art. Neuroscientist

duce how selected modes of intersensory perception

Hinderk Emrich stresses that for the investigation of

are stressed and visualized in artistic works. They are

synesthesia, the “phenomenon of transmodality” is

expressed either as transformation (sound and vision

important. Synesthesia means “creating processes” –

into space), as transliteration (text into imagination) or

and this is “never mono, always trans, inter, syn” (Em-

as translation (language into color codes).

53

so-called “form constants” (Heinrich Klüver). Klüver

sculpture with digitally projected moving images

did tests in 1920 on the effect of mescaline (peyote)

to create performing environments. Her work explores

consumption. Within his research of users’ individual

the interactions between physical space and moving

hallucinogenic reactions (very often seeing bright

images, and often uses technology and digital media

flickering lights), he called those geometric

as well as taking inspiration from immersive theater

patterns “form constants” and defined four groups:

and experimental film. Boyd’s work focuses on per-

(I) tunnels and funnels, (II) spirals, (III) lattices,

ception and the brain. Her recent complex project

including honeycombs and triangles, and (IV) cob-

The Point of Perception (Fig. 1) from 2009/13 explores

webs (Bressloff et al. 2002). Current researchers have

how much information the human brain needs in

gone further and hypothesized that these patterns

order to know what it is looking at. It was produced

could be a visualization of the brain’s structure: “In

in collaboration with neuroscientists Dr. Mark Lythgoe

general the images do not move with the eyes. We

and Dr. Beau Lotto from University College London,

interpret this to mean that they are generated in

and is both an art experience and a scientific arena

the brain. Here, we summarize a theory of their origin

on vision.

in visual cortex (area V1), based on the assumption

The installation uses moving images and built

that the form of the retino–cortical map and

environments to confuse depth perception and is

the architecture of V1 determine their geometry”

designed to act specifically on the human eye and

(Bressloff et al. 2002). So “form constants” can mirror

brain to create a space of uncertainty. Madi Boyd

the brain’s inner architecture during times of psycho-

describes her artistic work as combining “constructed

logical pressure – not only drug consumption, but

environments and projected films to create immersive

also mental stress such as sensory deprivation. For

installations investigating the interaction between

Boyd it was an important part of her artistic process

moving image, space, and the brain.” She sees “the

as it helped her to understand how the elements she

screen as sculpture and light as paint on the canvas

worked with (motion, light, pattern, form, space) were

of dark space” (Boyd 2013). A newly developed version of the project was presented at Art Laboratory Berlin in 2013 with the addition of sound and color, bringing another parameter of transmodality into the art piece. The

received and interpreted by the brain, allowing Boyd to better predict how someone might perceive and experience her work. The process of spatial transformation within this installation is remarkable: The use of the endless

idea was to assign musical notes and colors to

loop gives it a never-ending, almost hallucinogenic

specific points in the gridded space. The sound was

effect. Madi Boyd’s piece is a perfect example of the

created by composer Nye Parry, and was composed in

so-called filled-space installations: Enchantments,

a way described by Madi Boyd, who herself perceives

where there is “an overall environment with little or

a form of color-sound synesthesia. Boyd challenges

no escape route, the enchantment draws heavily

the viewer’s spatial perception with a complex video

on their theatrical roots, the suspension of disbelief

projection of geometric structures in a dark room.

being chief among these” (Rosentahl 2003). Taking

The slow movements of the work’s projected abstract

into consideration the theory of the “form constants”

forms not only match the ambient sound that refers

as the visualized architecture of the brain and the sus-

to the artist’s personal synesthetic perception. Boyd

pension of disbelief – one might argue that with this

also manages to visualize geometric patterns run-

installation piece we can experience a walk into our

ning across the installation’s dark walls and ceilings:

own, or even the artist’s, brain.

THEORIES ON DIGITAL SYNESTHESIA

54

Transformation, Transliteration and Translation. Synesthesia and Multisensory Perception in Contemporary Visual Art

Madi Boyd is an installation artist, fusing large-scale

Regine Rapp

1. Madi Boyd – Synesthesia as Spatial Transformation

2. Ditte Lyngkær Pedersen – Digital Synesthesia as Transliteration Ditte Lyngkær Pedersen is a multidisciplinary artist

actually.” There is also protagonist Emily’s statement,

working in video and installation, and whose work

where we perceive the particular power the sound of

often includes collaborators. She has invested in

words can have on synesthetes (quite physically and

deconstructing the intersections of subjective and

literally): “PURGATORY, when I hear the word, I see

objective perspectives, in order to pose alternative

a brown-green pile of feathery shapes, sticky to touch,

narratives and forms of representation. Her work

with leaping flames of liquid red. It tastes like a sweet

critically explores the dynamics of language,

cheese on bitter greens wrapped in an apple with

synesthesia, and identity. Her long-term art project

lemon squeezed on; shaped like an abstract figure

on synesthesia Why is Green a Red Word? began

of metal; life-size, that I feel by sticking my arms into

in 2003 and triggered her choice of incorporating

it and running my fingers along its unpredictable

explorations on synesthesia into her art practice.

turns. […]” (Lyngkær Pedersen 2013).

Since 2003 Pedersen has created an extensive

Pedersen chose a fascinating method of transliter-

video archive of interviews about the multi-sensory

ation between word and image (as in the French word

perception of synesthesia that documents the expe-

“image”): Although no speaker is visually depicted in

riences of individuals and at the same time makes the

the video, it is nevertheless her voice that talks about

unbridgeable gap between this topic and the audience

the individual association. The recipient is mesmerized

clear. Her project Why Is Green a Red Word? is com-

by the distinct spoken, but invisible, explanations – on

prised of interviews with synesthetes and scientists.

the screen there is nothing but the static, motion- and

Pedersen’s long term project includes several

colorless letters of the word “purgatory”. Pedersen

outstanding video pieces – for example, the concep-

convincingly achieves the moment of “creating pro-

tual video work What the Hell does Purgatory Look

cesses” (Emrich 2013). Her chosen mode of transliter-

Like? (Fig. 2). Although the monitor shows throughout

ation is full of “trans”, “inter”, and “syn” as the listener

its entire length the word “purgatory” in large black

can’t help but follow the rich lively explanations of the

letters on a white background, the recipient (viewer)

protagonists. This trans-literation, remarkably enough,

receives the main impulse via headphones: several

leads us away from semantic meaning.

voices talk one after the other about their individual synesthetic perception of the word purgatory. Most of the time, however, they refer to the visual characterization; the semantics are often secondary or even undermined. Selected answers by the protagonists show us a plasticity, and an asemantic, sound-oriented quality. Protagonist Karen states: “PURGATORY, resembles a prune, but the word is partly greasy like Vaseline Petroleum Jelly. The syllables account for this. The word ‘group‘ is also like Vaseline. You can spell ‘group‘ from the letters that make up Purgatory.” Another protagonist, Anne Constance Cooper, notes: “PURGATORY, is the colors of a day old bruise; purple and blue and black. The color has a slightly wavering, heavy sound, like the big 4th note of Beethoven’s 5th. Interestingly, it is a less pleasant word than hell,

55

3. Eva-Maria Bolz – Digital Synesthesia as Translation Eva-Maria Bolz is an artist, graphic designer and

synesthetic coloring for letters, numbers and whole

grapheme synesthete based in Berlin. In the center

words. With the help of an algorithm, selected texts

of her creative activity is the intensive study of

have been converted into large format color plates.

typography and its implementation. Bolz has based

These plates consist of thousands of color bars and

her abstract color system directly to her own graph-

include all the information of the text. For Bolz “the

eme synesthetic perception: For many years she has

texts no longer speak about the form of the grapheme;

collected and recorded her own relationship between

instead the content is transmitted exclusively through

text and color in the form of a digital diary. She has

the associated synesthetic colors“ (Boltz 2013). Here the artist undermines the stressed semantics and system, as in the color panels we lose the sense

computer to compare slight changes in perception

of the content but consider the piece as an “all-over”

Her work Der innere Monitor (The Inner Monitor)

abstract. To speak in semiotic terms: Here the signifier becomes the signified and vice versa – the content

is dedicated to an exploration of the relationship

overlaps with the form. There is an interesting dualistic

between color, text and visual perception. Due to

aesthetic in Bolz’ aesthetic translation project: The

her grapheme synesthesia, she feels an unchanging

work Der innere Monitor appears as minimal due to

association of colors to numbers, letters, as well as

its strict systematic outline, but also as highly

selected whole words. Perception becomes a filter

playfully poetic, due to her choice of literary text. The

through which letters, words – text in itself – are

strict rules of translating letters into color (grapheme

translated into colors and transformed from a set of

synesthesia) oppose the humorous tone of the

well-known characters into a message that can be

work, echoing the wit in Ionesco’s play with nonsense

detected by means of a particular synesthetic

and grotesquery. We finally feel challenged by the

sensibility. Her project Der Innere Monitor follows her

opposition of dry linguistic phonetic sentences on

perception that colors and letters form a specific

the one hand, and the intimacy of the artist’s own

code through which a text can be translated into

inner color system on the other: by means of a

blocks of color. Each letter corresponds to a specific

code, the artist’s individual visual system has been

color. The artist deliberately uses texts that contain

translated into plates of color.

intense color descriptions, such as Oscar Wilde’s The Nightingale and The Rose. In the exhibition in spring 2013 at Art Laboratory Berlin, Bolz presented five selected texts in the form of large color plates with texts by Oscar Wilde, E. T. A  Hofmann, Joris-Karl Huysmans, Eugène Ionesco and Marie-Luise Kaschnitz. For Bolz, the work “is based on the hypothesis that grapheme, and in particular the lexical synesthesia, enables the visualization of text content for a fixed system of meaning. This is also based on individual experiences, but is constant and involuntary, so is not an expression of current metaphorical allocation“ (Boltz 2013). The specific method is noteworthy: over several years she has systematically recorded her

THEORIES ON DIGITAL SYNESTHESIA

56

Regine Rapp

and generate a continuity of color-text relationship.

Transformation, Transliteration and Translation. Synesthesia and Multisensory Perception in Contemporary Visual Art

regularly mixed the particular color tones of letters, words, days of the weeks or months over time at her

A

A Ditte Lyngkær Pedersen, What The Hell Does Purgatory Look Like?, 2005, Video, 8 Min., exhibition view, Art Laboratory Berlin, 2013 | Installation view, Art Laboratory Berlin, 2013 Photo: Tim Deussen, 2013

B Madi Boyd, The Point of Perception, 2009/13, installation view, Art Laboratory Berlin, 2013 | Installation view, Art Laboratory Berlin, 2013 Photo: Amin Akhtar, 2013

B

57

Boyd, Madi. 2013. “The Point of Perception. On Collaborations Between Artists and Neuroscientists.” Presentation at Art Laboratory Berlin’s conference Synaesthesia.
 Discussing a Phenomenon in the Arts, Humanities and (Neuro-)Science, Berlin, July 2013. Accessed January 30, 2016. http://artlaboratory-berlin. org/html/de-event-17.htm Bolz, Eva-Maria. 2013. In conversation with Regine Rapp. See also

exhibition text - Accessed January 30, 2016: http://artlaboratory-berlin. org/assets/pdf/Syn%204_Pressetext_2013_5_15.pdf

(Neuro-)Science, Berlin, July 2013. Accessed January 30, 2016. http://artlaboratory-berlin.org/html/ de-event-17.htm

Bressloff, Paul C. et al. 2002. “What Geometric Visual Hallucinations Tell Us About the Visual Cortex.” Neural Computation, The MIT Press, 14 (3): 473–491.

Lyngkær Pedersen, Ditte. 2013. Why is Green a Red Word? Artist Book. Published by Art Laboratory Berlin

Emrich, Hinderk. 2013. “Synaesthesia, Synaisthesis and the Enhancement of Coherence.” Keynote speech at Synaesthesia. Discussing a Phenomenon in the Arts, Humanities and

THEORIES ON DIGITAL SYNESTHESIA

Rosentahl, Mark. 2003. Understanding installation Art. From Duchamp to Holzer. New York München: Prestel. 33. Trautmann-Lengsfeld, Sina A. 2013. “Multisensory Processing in

58

Synaesthesia.” Presentation at Art Laboratory Berlin’s conference Synaesthesia.
Discussing a Phenomenon in the Arts, Humanities and (Neuro-)Science, Berlin, July 2013. Accessed January 30, 2016. http://artlaboratory-berlin.org/html/ de-event-17.htm

Transformation, Transliteration and Translation. Synesthesia and Multisensory Perception in Contemporary Visual Art

References

Regine Rapp

C Eva-Maria Bolz, Der innere Monitor / Oscar Wilde, 2013, Digital printinstallation view, Art Laboratory Berlin, 2013

Sensing Digital Art: Aesthetic Acts Beyond (the two) Synesthesia(s)

Chris Salter

  Keywords   touch, digital art, immersive art, multisensory integration, sensory substitution, stimuli, sensation, perception

It seems that everywhere we look, hear and feel in

sense-based information” and that haptics in

contemporary art and life, digital computational

particular would enable us to “feel the surface of

technologies are quickly overtaking our senses;

produce”, or eventually “virtually” high-five a

particularly our sense of touch. While Apple laptops,

hologram of Tupac Shakur (Schwartz 2012). More

for example, have long been adorned with seeing

recently, a NY Times feature on “The Future of Touch”

through their “retinal” displays , they have more

argued that “to interact with the world in any mean-

recently began to employ “Force Touch” and “Taptic

ingful way, we have to use the sense of touch” and

engines,” the latter of which aims to transform the

that haptic technologies would be the breakthrough

body itself into an interface.

in letting “people feel things that are not actually

In 2012, IBM’s “cognitive computing” research program’s “5 in 5” argued that in five years, machines could “extend our ability to gather and process

there” (Fergusson and Naudziunas 2015). I am zeroing in on touch here not simply because it appears to be the latest sense modality to become

59

the subject of (and subject to) technological buzz and

view, mainly focusing on two sense modalities in

hype. Touch, after all, has long been viewed as a

relationship to another . In Spence, Calvert and Stein’s

forgotten sense; one that as cultural historian

monumental volume on multisensory processes

Constance Classen claims “often remains unspoken

(2004), for example, the vast majority of articles ex-

and, even more so, un-historicized” (Classen xi, 2012).

amine two modalities crossing over each other. The

Increasingly integrated into phones, laptops, game

fabled McGurk effect, in which vision alters speech

controllers, theater and car seats, robotic arms

perception when the auditory component of one

and elsewhere, it seems that the science of haptics,

sound is paired with the visual component of another

applying force to the skin to deliver feedback and

sound, leading to the perception of a third (hearing ba

information, is now on the verge of becoming a new

and seeing ga results in perceiving da), involves vision and audition. Likewise, the pioneering cross-modal

Yet, touch is also interesting because it is one of

research from neuroscientist Shinsuke Shimojo also

those strange sensory modalities that appear to “cross

focused on only two modalities, namely, how sound

over” or “interfere” with the other senses. As a field

can alter the temporal and spatial structure of vision

of vibration, touch can suddenly be perceived as

(Spence, Calvert and Stein 2004, 27– 34). The notion of sensory substitution, made famous by Mexican neuroscientist Paul Bach y Rita’s work

audible range. Even Aristotle (2010), who is blamed

on brain plasticity in the 1960s (Bach-y-Rita, 1969)

for the classic reduction of sense perception into

likewise focused mainly on two sense modalities. In

five discrete categories, acknowledged that the neat

sensory substitution, one human sense is used to

definitions of organ to phenomena (correspondences

receive information normally associated with another

between sense organs and things sensed) was not

sense, for example, Bach-y-Rita’s TVS (Tactile Vision

so simple, arguing that “if touch is not one sense but

Substitution) in which a matrix of electro-tactile stim-

many,” then the fact that the “tangible sensibles are

ulators works to convey spatial information normally

also many … involves perplexity.” Thus, although it

associated with vision to the skin. Now a commercial

seems to occupy the surface of the skin, touch for

industry of assistive devices, sensory substitution

Aristotle may also be internal, involving many

usually implements the substitution of the visual sense

opposing (yet binary) sensations such as hot/cold,

in a machine through either auditory or tactile senses,

wet/dry, hard/soft and so on (2010, 69).

but rarely using more than two modalities.

In the social and natural sciences, however, it seems that any strange aberration of sense perception,

Even our current obsession with the almost mythical notion of synesthesia seemingly follows

like Aristotle’s understanding of touch, is perplexing

this dualistic pattern. The Oxford Handbook of

since it does not neatly fall into one modality and

Synesthesia defines synesthesia as “a neurological

always follows dualistic, dyadic patterns of vision/

condition which gives rise to extraordinary sensations”

hearing, touch/hearing and others. Many studies of

(Simner and Hubbard 2013, xxi), in which the “stim-

so-called multisensory integration, defined by Stein

ulation of one sense modality automatically evokes

and Meredith as a “statistically significant difference

a perception in an unstimulated modality” (2013, 3).

between the number of impulses that are evoked by

Even though synesthesia describes the “merging of

a cross-modal combination of stimuli and the

the senses,” a majority of researchers see this merg-

number of impulses evoked by the most effective

ing in pairs where an “inducer” in one sense triggers a

of these stimuli individually” (Calvert, Spence and Stein

consistent concurrent sensation in another modality:

2004, 248) purportedly claim to critique the “sense

color to grapheme (the most common form of

by sense” approach of traditional perceptual

synesthesia), mirror touch or lexical-gustatory (the

psychology and neuroscience. But most of this

rarest in which associations are formed between

research still emphasizes a dualistic sensory world-

words and tastes), among dozens of others.

60

Chris Salter

acoustic when it enters into the limited range of human hearing, moving from below 20 Hz into the

Sensing Digital Art: Aesthetic Acts Beyond (the two) Synesthesia(s)

socio-cultural paradigm.

But as anthropologist of the senses David Howes

synesthesia through what is still mainly a history of

reminds us (2013), synesthesia also operates within

audio-visual correspondences, from color organs to

another duality. Neuroscience, which is enjoying a

the Whitney-inspired slit-screen effects set to Ligeti’s

kind of pre-eminent status among the sciences at

Atmospheres and Requiem in the penultimate

the moment, emphatically distinguishes between

psychedelic conclusion of Kubrick’s 2001.

two forms of synesthesia: so-called “strong” synesthesia, defined as the brain-based, mainly congenital

In fact, as we dig deeper, artistic events that use the senses as material for the production of “blocs of

or “genuine, developmental” type in which subjects

percepts, sensations and affects” (Deleuze and

report synesthesia for as long as they remember, and

Guattari 1994, 167) more often than not transcend

“weak” synesthesia, which describes “milder forms of

such sensory dualisms. They introduce “perplexity”

cross-sensory connections revealed through language

into what Howes identifies as worlds “beyond and

and perception” (Martino, et al. 2001, 61).

beneath the five senses.” Beyond the five senses

For Howes, such weak synesthesia is not weak

invokes aesthetic experimentation into phenomena

at all, but rather constitutes the notion not only as

like the supernatural and the paranormal, while

a neurological practice but also as a cultural one.

“beneath the five senses” indicates the worlds of

Indeed, cultural practices that involve cross-modality,

“the visceral and the molecular” (Howes 2009, 22).

such as the ritual use of hallucinogens by the Desana

Indeed, as the 20th century theater sorcerer Antonin

and Tucano Indians of the Northern Colombian

Artaud wrote, a theater of cruelty would be one that

Amazon region, already articulate a merge between

is “addressed to the entire organism,” in which tech-

the biological, social-technical and cultural

nologies like light, sound and other scenographic

(Howes 2013, 156 – 159).

elements would produce “sensations of heat, cold,

Given that this volume focuses on current artistic practices that explore the intersections among synes-

anger, fear, etc.” (Artaud 1958, 95). The binary, switch-like nature of the digital also

thesia, digital art and aesthetics, it would seem that a

participates in the propagation of such sensorial

discussion of why the notion of synesthesia has played

dualism, giving short shrift to the complex entangle-

such a historically significant role in art practice would

ments that take place in the interdependent processes

also be in order. In fact, much has been written about

of stimuli, sensation and perception. That artists

the cultural synesthetic practices of artists as far-

increasingly employ digital technologies that repre-

ranging as Baudelaire, Scriabin, Kandinsky, Varèse,

sent stimulations and sensations in binary numbers

John Whitney or Gyorgy Ligeti, in which their interests

and then re-render them in strictly analog (that is,

in synesthetic processes catalyzed the development

analogous) form from those binary numbers in no way

of entirely new artistic styles and movements (van

implies that such processes can be reduced down to

Campen 2013, 631– 632). As is the case with touch,

simply numerical representations.

these histories have equally been silenced, with art

As Cretien van Campen argues, the immersive

historian Simon Shaw-Miller arguing that synesthesia

nature of contemporary art operates not just in

is denied by an art-historical frame in which “the eye is

sense categories, but on the body itself (van Campen

mute and the ear cannot see” (Shaw-Miller 2013,1).

2013, 635). The use of pressure, temperature,

Shaw-Miller is one of a handful of art historians

tension, proprioception, nociception and other felt

aiming to correct the academic tendency to divide

sensations by artists that combine, mix, intensify

the senses up among different discursive formations:

and diminish over time, generating unknown and

disciplines and faculties which split up art history from

uncertain transmutations, densities, colors, rhythms

musicology, “the sounds of art and the sight of music”

and “zones of intensities” (Varèse 1966, 13) goes

(2013, 1). Yet upon closer observation (or sensation),

far beyond the dualistic nature of correspondence

artistic practices do not usually hold to even the

between vision and audition, vision and touch,

supposedly current revisionist art-historical focus on

smell and taste.

61

In their 1991 work Divina Commedia: Praxis for

beyond the synesthetic evocation of “hearing shapes”

Death, for example, the Japanese artists/researchers

or “feeling vision,” and instead generate experiences

Masayuki Towata and Yasuaki Matsumoto designed

of a threshold nature in which sounds, light and

a completely bodily immersive installation in which

the thickness of space are felt moving around the

visitors would don specially designed suits and, while

skin and the shape of perceived time and space are

floating inside a 30 cm deep pool of edible gel (Iwaki

twisted inside out. Thus, despite its long (and still

2013, 217), experience a choreographed sequence of

ignored) historical trajectory, artistic work operating

flashing lights, generated from an array of computer-

at the borders between new technologies and new

controlled Dataflash strobes hung directly above.

morphologies of sensation might do well to bypass

Aiming to “loosen their sensory-motor connections,”

its supposedly radical synesthetic genealogies, in-

Towata and Matsumoto’s work was inspired by the

stead embracing Artaud’s challenge to create art as a

float tanks of John Lilly as well as research into

spectacle “immersed in a profound intoxication which

near-death experiences, a term coined by the

restores to us the very elements of ecstasy” (1958, 65).

American medical doctor Raymond Moody. Although this installation which explored the “grey zone between death and life” utilized both acoustic and visual media, we would be hard-pressed to describe Chris Salter

the almost out-of-body experiences and nearthreshold effects experienced by audience members (one of whom claimed to have “met Buddha” during the experience) as merely synesthetic (2013, 219). Later artistic works like artist Kurt Hentschläger’s ganzfeld-inspired installation Zee (2008) — in

Sensing Digital Art: Aesthetic Acts Beyond (the two) Synesthesia(s)

which audiences are plunged into blinding fields of stroboscopic light and impenetrable fog — or Sonia Cilari’s audience-performer intensive Sensitive to Pleasure (2010), in which the participant is caught up in near darkness, interacting with an indistinguishable body inside a room that, unbeknownst to them, results in the triggering of mild but persistent electrical pulses on the artists’ arms and chest outside of the space (http://nimk.nl/blog/sensitivetopleasure/) also go beyond the dualistic notions of synesthesia, instead generating full-body affects in which the corporeal sense of space and time are radically and inexplicably altered. Similarly, our performative environment Ilinx (Lamontagne, et al. 2015) where audience members wear wirelessly enabled, full-body haptic clothing inside an audio-visual environment which blurs together the modalities of sight, sound and touch, also results in a panoply of sensory impressions that generate simultaneous proprioceptive, haptic, acoustic, (semi)-visual, equilibrioceptive and temporal feelings and affects. These zones of intensities go

THEORIES ON DIGITAL SYNESTHESIA

62

A

B

A Sonia Cillari: Sensitive to Pleasure (2010–2011) Performative electric-field sensing and sound environment Photo: Sara Tirelli

B Kurt Hentschläger: ZEE (2008) Immersive audiovisual environment Photo: Kurt Hentschläger

63

Aristotle. 2010. De Anima. Trans. Mark Shiffman. New York: Focus Philosophical Library. Artaud, Antonin. The Theater and its Double. Trans. M.C. Richards. New York: Grove Press, 1958. Bach-Y-Rita, Paul, et al. 1969. “Vision substitution by tactile image projection.” Nature 221: 963 –964. Calvert, Gemma, Charles Spence and Barry Stein. 2004. Handbook of Multisensory Processes. Cambridge, MA: MIT Press. Cillari, Sonia. 2010. http:// www.soniacillari.net/Sensitive_ to_Pleasure.htm. (Accessed November 2, 2015)

Classen, Constance. 2012. The Deepest Sense: A cultural history of touch. Champagne-Urbana: University of Illinois Press. Deleuze, Gilles, and Félix Guattari.  1994. What is Philosophy? New York: Columbia University Press. Fergusson, Rebekah and Jessica Naudziunas. September 9, 2015. “The Future of Touch.” Nytimes.org. Accessed November 1, 2015) Hentschläger, Kurt. 2008. http://kurthentschlager.com/ portfolio/zee/zee.html. (Accessed November 1, 2015) Howes, David. 2009. The Sixth Sense Reader. Oxford: Berg Publishers. Howes, David, and Constance Classen. 2013. Ways of Sensing:

THEORIES ON DIGITAL SYNESTHESIA

Understanding the Senses in Society. London: Routledge. Martino, Gail, and Lawrence E. Marks. 2001. “Synesthesia: Strong and weak.” Current Directions in Psychological Science 10.2: 61–65. Iwaki, Akihisa. 2013. “The Body as an Image Processor.” In Diversities in Aesthetics: Selected Papers of the 18th Congress of International Aesthetics, edited by Peng, Gao Jian Ping and Peng Feng. 213–223. Beijing: China Social Sciences Press. Lamontagne, Valerie, et al. 2015. “The Ilinx Garments: Whole Body Tactile Experience in a Multisensorial Art Installation.” Proceedings of the International Symposium on Electronic Art (ISEA), Vancouver, 2015.

64

Schwartz, Robin. 2012. “IBM 5 in 5 2012: Touch.” Accessed November 2, 2015. http://ibmresearchnews. blogspot.ca/2012/12/ibm-5-in-52012-touch.html. Shaw-Miller, Simon. 2013. Eye hEar The Visual in Music. London: Ashgate. Simner, Julia, and Edward M. Hubbard, eds. 2013. Oxford Handbook of Synesthesia. Oxford: Oxford University Press. van Campen, Cretien. 2013. “Synesthesia in the Visual Arts.” In The Oxford Handbook of Synesthesia, edited by Julia Simner and Edward M. Hubbard. 631–646. Oxford: Oxford University Press. Varèse, Edgard, and Chou Wen-Chung. 1966. “The Liberation of Sound.”  Perspectives of New Music: 11–19.

Sensing Digital Art: Aesthetic Acts Beyond (the two) Synesthesia(s)

References

Chris Salter

C Ilinx, 2014/2015 (Performance environment) © Chris Salter

Digital Synesthesia: Reminiscing about Science and Art

Romana K. Schuler

  Keywords   psychophysics, organ projection, physical apparatus, seeing apparatus, experimental physiology and psychology, multimediality, expanded art, expanded cinema, digital art

What do we envision when we think of the new term

everyone is born with these abilities, or whether they

digital synesthesia? What does it mean to attempt to

can be learned, has become a hot topic in memory

integrate the term digital synesthesia into the context

research over the past few decades.

of art and art history? Synesthesia takes on a different meaning when

And yet the question arises as to what digital synesthesia has to do with the original meaning of

combined with the adjective digital. The classic term

synesthesia? Strictly speaking, nothing. Digital stems

synesthesia stems from 19th century conception

from the Latin word meaning digitus, the finger. As of

physiology — as the description for a very specific

the mid-17th century the term is mentioned for the

ability of the sensory organs and sensory perceptions;

first time in connection with fingers (Behrens 2014).

that is the linking of physically separate cognitive do-

The use of the term digital prevailed in connection

mains (fields). How synesthesia develops and whether

with computer technology in the mid-20th century.

65

Today, digital is a buzzword and synonymous with

to expand upon the senses already seems to be upon

modern computer systems. Generally though,

us, to some degree, with the complex computer

the term remains especially vague except for its

systems of today. The goal being aspired towards is

specific use in information technology. In connection

the perfect duplication of the human form. The

with the term synesthesia it points to a current

next step would then be to breathe digitized sensory

problem, which is how can rarely occurring cognitive

perception into this artificial apparatus. From an

processes, that only synesthetitians experience, be

anthropological standpoint, this idea of organ

expressed digitally with digital tools?

projection (Knapp 1877, 27) is age-old and probably

For some time now, it has been clear that

began with the invention of the plow, which was

modern computer science experiments with the

the first complex tool developed by humans

simulation of sensory perception. The potential

(Knapp 1877, 76).

Computer and math: I believe the computer metaphor still has a lot to teach us about the brain. I am particularly interested in brain analogs of operating systems — that are illuminating for cognitive psychology subjects, such as task switching. I also have a long-standing interest in geometry as a constructive theory of perception.

experiments in visual perception (1862 – 1868, Mach). Aside from this fundamental research – the role as pioneer in the study of the human body – his work also led to a mechanical form of expression that appeared to be sufficient for the human body in science. Natural scientists consider the body to

Romana K. Schuler

Sensory Perception: A Mathematical Problem?

be an apparatus, for example an optical apparatus.

(Shabtai Barash 2006)

Even in Freud’s writings we can find that the expres-

The first scientific foundation of the phenomena of

often used.

sion “psychischer Apparat” (mental apparatus) is A main feature of the new sciences in the 19th

can be found in 19th century perception research.

century was the emphasis on the experimental

At that time, the study of human perception did not

method. The experimental procedure became the

generally take place in the physiological areas of

foundational principle of every scientific discipline.

natural sciences but chiefly by physicists and mathe-

The designation of the new specialization (area of

maticians who picked up on the topic and centered

study) was made evident through the title: experi-

their research on it. What was the reason for these

mental physics, experimental physiology and experi-

interdisciplinary tendencies amongst physicists?

mental psychology. This turning point in science was

During the 19th century, physics and mathematics

also actively propagated in society — a succession of

became independent scientific disciplines. Ernst

accomplished scientists held and published so-called

Mach reported that under duress, due to his financial

popular scientific speeches. Beyond that, cultural

situation, he was forced to work in the field of physiol-

changes took place brought on by their far-reaching

ogy after completing his studies in physics because

discoveries, and these resonate to this day. The

he simply could not afford the equipment necessary

insights of the new sciences were picked up and

for studying physics. A second important impulse for

reflected upon with enthusiasm by artists and authors

him was the publication of G.T. Fechner’s Psycho-

alike. In turn, personalities in the sciences such as

physik (1860). Fechner’s approach, as presented in the

Helmholtz or Mach also spoke about visual art in

publication, encouraged Mach to hold a semester-

connection with their new theories on sensory

long lecture on the subject at the University of Graz,

perception. Helmholtz is said to have influenced the

Austria. A short time later he undertook his own

French Impressionists with his optical studies. And

THEORIES ON DIGITAL SYNESTHESIA

66

Digital Synesthesia: Reminiscing about Science and Art

synesthesia and the precursor of digital synesthesia

Mach influenced abstract painting and media artists

were later termed “Chladni’s Klangfiguren” (Chladni’s

such as Dan Graham and Peter Weibel (Schuler 2014).

Figures or Sound Figures). Chladni’s experiments in

Although the previous perception researchers, such as Jan Evangelista Purkinje, David Brewster,

acoustics motivated Michael Faraday, Charles Wheatstone and Ernst Heinrich Weber to lead further

Joseph Plateau, Simon Stampfer, Charles Wheatstone,

experiments in the field. The discovery of sound

Theodor Gustav Fechner, Hermann Helmholtz, Ernst

figures attracted the interest of artists such as Philipp

Brücke and Ernst Mach were physicists or mathe-

Otto Runge, Caspar David Friedrich and Goethe;

maticians, they made considerable contributions to

the latter was also in contact with Chladni. In 1808,

the study of physiological sensory perception. The

Purkinje began to analyze these sound figures in

functions and perceptive capabilities of the human

more detail and in doing so, discovered very similar

body were studied intensively. In this fertile scientific

structures to those of the human eye (Purkinje 1824).

environment the first experiments studying synes-

He shared his discoveries and insights with

thesia took place. The artificial word “syn-ästhesie”

Goethe, among other people. He assumed that the

(syn-aesthesia) was coined in 1866 by the French

visual organ could recognize sound and figures and

neurologist Alfred Volpian. The term was not able

that the eye and ear could possibly match in their

to gain wide acceptance though. In general the

sense of perception.

appearance of this unusual perception was described

Human perception was supposed to be studied

as double or multiple sensations, or as “Farbenhören”

empirically using mathematical formulas and be

(hearing colors) because this was the commonest

experimentally repeatable. A higher-ranking central

form of synesthesia. This phenomenon, as the

research question during the 19th century was

eponym describes, had already been commented

the search for a common canon for the function of

on by many other scientists before him. The oldest

all the senses. Behind the idea was that by means of

account of a synesthetic perception can incidentally

employing a measurable method using repeatable

be found in the Old Testament. When the people

experiments, one would come across a constant.

of Israel were in the desert waiting, the voice of

With the help of this, in an ideal case, mathematical

God could be heard: “... and they saw the voices”

formulas could be deduced. For this purpose, one

(Exodus 20.14).

had to first study the functionality of all the receptors

This genuine synesthesia actually has very little

in terms of touch, taste and sense of smell, and

to do with digital synesthesia. And yet, on the one

especially in terms of the sense of hearing and the

hand genuine synesthesia has begun to be regarded

sense of sight. Each sense was looked at in its

in a decidedly scientific manner. On the other hand

individual components, measured and analyzed.

mathematical, algorithmic or “digital” synesthesia, as we would term it today, was quasi-invented in the 19th century. In this context, special attention should be paid to the very gifted Bohemian-born researcher, Jan

The transition from the scientific perspective of the early-19th-century physical approach to Fechner’s psycho-physics was initiated by Weber’s studies on the sense of touch. Both Weber and Fechner thought they recognized physical-psychological connections

Evangelista Purkinje, who at the turn of the 19th

for the first time. Subsequently, countless conclusions

century grappled with the after image, the kinetic

were drawn as to the skin’s constitution, culminating

movement as well as the phenomena of synesthesia

in a characterology of the skin. Pressure, touch or

among other things (Schuler [2012] 2016). In his

vibrations, the influence of cold and warm, the sense

dissertation dated 1819, Purkinje referred to the

of pain, etc. were studied.

physicist and lawyer Erwin Chladni’s (1787) sound

Fechner recognized a continuation of his own

figures (Fig. 1, 2). Chladni had thin plates strewn with

approach in Weber’s theorem dated 1834. Weber had

sand and set into oscillation with the help of bell-

set weights on his test persons and arbitrarily added

and tuning- forks. In doing so, figures emerged that

more weight in order to ascertain the relationship

67

warnings were sounded as to the one-sidedness of

Fechner was convinced he had found a viable

purely experimental psychology. The well-known

psychophysical fundamental law for all the senses.

Gestalt psychologist Egon Brunswick criticized the

He was able to put it on paper in a mathematical

methodological physicalism that more or less

formula: e = log r. The perception (e) increases

dominated the entire 19th century (Brunswick 1925).

proportionately to the logarithm of the stimulus (r)

At the same time, it was particularly physicists such

(Fechner 1889, 18). This equation became known as

as Mach, representing element-psychology (1886),

the Weber-Fechner law. In his comprehensive

who pointed out the differences between optical and

“Psychophysik” (psychophysics), Fechner was one

geometric similarities. Mach’s work had encouraged

of the first to comment on double- and multiple-

Christian von Ehrenfels in his Gestalt theory (1890)

sensations based on the example of the phenomenon

which was significantly represented in experimental

of hearing colors. The term “Experimentelle

psychology up until the 1940s. It was then superseded

Ästhetik” (experimental aesthetics) can be traced

by cognitive science and cybernetic theory.

Carl Stumpf was a pioneer in the field of experi-

It may well be that all these 19th century trials, deliberations and theories about sensory perception

mental sound- and tonal-cognition in the second

seem outdated. Yet they are the starting point for

half of the 19th century. Initially he was a student of

a field of study that we call digital synesthesia today:

the philosopher and perception researcher Franz

examining and measuring the functions of the

Brentano, and in 1894 Stumpf became professor

senses and sensations so that in the end, they can

of experimental psychology in Berlin. Amongst his

become a graspable algorithm. And so, steering

students were the founders of the famous Berlin

with these machines becomes possible; the machines

Gestalt theory – Max Wertheimer, Wolfgang Köhler,

can be equipped with exceptional features — especially

Kurt Koffka and Kurt Levin.

with regard to big data — namely operating in a

At the age of 22, Stumpf wrote his postdoctoral thesis Die Grundsätze der Mathematik (1870) (The

synesthetic manner. Just like psychophysics, the basis of digital

principles of mathematics) with Hermann Lotz.

synesthesia lies in its double meaning. Synesthesia

A short time thereafter he concentrated on the

implies that the modalities, the perception groups

concept of space and the perception of sound.

of the individual sensory sectors become evident in

He introduced the terms “concordance” and

psychologically different ways.

“discordance” in his Tonpsychologie (1874) (Sound

The wish to rebuild nature in a technological

psychology). Deeply impressed by Weber’s and

manner seems to be an elemental impulse in science.

Fechner’s work, Stumpf began to study inter-

Since the 17th century there has been a desire to

disciplinarily. His greatest achievements can most

dehumanize and reproduce the human spirit in the

notably be found in the field of acoustic perception.

form of machines. At the turn of the 20th century,

He connected these with his phenomenological

machines were used in industry that were equipped

ideas and was able to come to new insights. Around

with simple “Sinneswahrnehmungen” (sensory

1900, with his assistant Erich Hornbostel, Stumpf

perceptions) such as an automatic stop when there

founded the phonogram-archive that is part of the

was a malfunction (Moravec 1990, 22).

Ethnological Museum in Berlin. With his phenomeno-

To date, we remain infected by this history of

logical theory he also influenced Edmund Husserl

thought. Artistic works that evolve in terms of

in the 20th century.

digital synesthesia are tied to this idea.

Only at the turn of the century was the so-called Wissenschaft der Seele (science of the soul) able to be established in psychology. At that time psychology had an experimental orientation. Soon thereafter,

THEORIES ON DIGITAL SYNESTHESIA

Romana K. Schuler

back to Fechner.

68

Digital Synesthesia: Reminiscing about Science and Art

between sensation intensity and stimulus intensity.

Fig. 1

Fig. 1, 2 Chladni’s Sound Figures, 1789 (author’s archive)

Fig. 2

69

Digital Synesthesia and Art Since the beginning of the 20th century psychologists

(Oops we are alive) in his political theater in Berlin. A

have also studied and published on synesthesia to a

four story high stage was created in order to play short

great extent. The psychologist Annelies Argelander

scenes on it as well as film sequences on a screen that

aired her extensive research in the publication Das

had been placed at the center. Piscator worked with

Farbenhören und der synästhetische Faktor der

authors such as Bertold Brecht and Heinrich Mann

Wahrnehmung (Color hearing and the synthetic factor

as well as with artists such as George Grozs, John

of perception) at the Munich Psychology Congress

Heartfield and László Moholy-Nagy on the stage set.

Argelander listed 466 bibliographical references. The writings were printed between 1800 and 1927 and are

their film as “expanding cinema” (Bute and Nemetz

an impressive documentation of the interest in the

1936). Bute, who had come from a painting back-

phenomenon. In 1927 the well-known synesthesia re-

ground, experimented in her animations with synes-

searcher Georg Anschütz organized the first congress

thetic effects in a similar fashion to Oskar Fischinger.

on color-tone-research in Hamburg. An interest in

She used the stroboscope to create effects that she

synesthesia was also reflected in the liberal arts and

called “seeing sound”. In the 1940s the first basic writ-

cultural research from this period. In his speech Die

ings on cybernetics and information theory were pub-

Urgeschichte der Synästhesie und ihre Ausbreitung im

lished. Norbert Wiener’s Cybernetics or Control and

älteren Geistesleben (The prehistory of synesthesia

Communication in the Animal and the Machine came

and its spread in older spiritual life), the Viennese

out in 1948 and C E Shannon published his famous

psychologist Albert Wellek emphasized the relevance

essay The Mathematical Theory of Communication.

of synesthesia in the liberal arts for the first time at a

Wiener was convinced that through technology,

congress on color-tone-studies. One year later Wellek

prostheses for human limbs and sensory organs

completed his doctoral studies with Doppelempfind-

would be developed or their performance would be

ungen und Programmusik (Double sensations and

enhanced. In The Human use of Human Beings (Cy-

program music) (1928). He then moved to Hamburg to

bernetics and Society) from 1950, Wiener refers to art

do research with Anschütz.

and literature as communication media that could not

At the same time, artists became interested in

remain untouched by the up-and-coming communi-

the synesthetic phenomenon. The influence of

cation technology or mass media. In this respect there

synesthetic appearances was represented in musical

was a backlog. He wrote: “History has shown that the

performances (Skrjabin, Lászlo) and in the emerging

development of natural sciences and art generally

medium of film (Ruttmann, Fischinger). In 1915

run parallel to one another. And yet, in the elapsed

Skrjabin, who was a synesthetician, presented his color

half-century, art has evolved far away from science.”

piano to a broad audience in 1915. In the summer

(Wiener 1952, 133) Wiener advocated for an interdis-

of 1925 a concert took place in Kiel (Germany).

ciplinary education; in which artists and intellectuals

Alexander Lásló performed on his further-developed

should not isolate themselves but show more interest

color piano; the instrument was specially built by

in interdisciplinary education (Wiener 1952, 148 – 149).

Ernemann-Werken. His Farblichtmusik (color light

Cybernetics and Information theory actually have

music) was performed about 1000 times in Germany

an effect on artistic work. Whether this is the result of

from 1925 – 27. There were numerous write ups on

a reaction to Wiener’s plea for engineering or for other

the shows and in total more than 40,000 people

reasons is anyone’s guess. While in applied art only

came to experience synesthesia. In 1927 the avant-

color, stone and wood were understood as relevant

garde producer Erwin Piscator directed a complex

material, art forms were being expanded upon with

media stage set for the piece Hoppla wir leben

light (op art) and movement (kinetic art) as well as

THEORIES ON DIGITAL SYNESTHESIA

Romana K. Schuler

In 1936, in an interview, the experimental filmmakers Mary Ellen Bute and Ted Nemetz described

70

Digital Synesthesia: Reminiscing about Science and Art

in 1925. In her 1927 publication with the same title,

sound (sound art). The first computers were rapidly

Kostelanetz who also created the term “mixed-means”

being used. Technical implementation was accepted

(Kostelanetz 1968). In 1964 – 65 the members of the

as artistic media although in the beginning the

American art group USCO (Gerd Stern, Steven Dunkee,

exorbitant costs only allowed for ideas and concepts.

Michael Callahan among others) settled down with

The notion of working with new media began to

their families in an abandoned church in Garnerville

appear in artistic activities. Expanding the concept of

(NY). They were artists, musicians, poets and techini-

the image, and ever more often a complete withdrawal

cians. The basic foci of the group were the theories

from the image, began with the spread of information

of new communication media — Marshall McLuhan

theory and cybernetics. The artist and mathematician

was one of the most important impulse generators.

Benjamin Laposky presented the first computer

During a meeting McLuhan advised the USCO group

drawings in the 1950s. The artist and computer pro-

to “Disregard the content and concentrate upon the

grammer Ken Knowlton developed the computer

effect” (Kuo/Callahan, 2008). In the same interview

program “Beflix” in 1963. With it, he and Stan Van Der

Callahan explained that McLuhan was a scientist and

Beek developed their famous geometric pixel images

that USCO had used science.

such as poem fields at Bell Telephone Labs in New

The influence of consciousness expansion was the

Jersey. The fascination for creating art with a machine

defined goal of the USCO group. According to them,

or with the help of a computer program appeared to

the experimentation with drugs and the use of new

splendidly confirm technology-oriented philosophy

image technology created the psychedelic environ-

as described by Knapp in the 19th century, especially

ment that the members of the group had imagined.

the concept of organ projection.

They and many others understood “psychadelic” as

With the invention of modern computer science,

synonymous with synesthesia. The young movie critic

a breakthrough took place that for the first time

Gene Youngblood, who tried to introduce the term

allowed and enabled the realization of synesthetic

synaesthetic cinema as a new concept of expanded

properties with the help of new technologies. Because

cinema, is an example. Characteristic of synaesthetic

this concept is a de facto algorithm it can be described

cinema was the “simultaneous perception of

as digital synesthesia. This creation of immersive tech-

harmonic opposites” according to the definition

nical environments in order to sound out new limits

(Youngblood 1970, 81).

in perception was tackled by several avant-garde artists

Synaesthetic cinema is neither subjective, nor

at the beginning of the 1960s. Multi-media events

objective, nor non-objective. It is a combination of

solicited a great deal of attention. The architects

everything; what Youngblood calls an extra-objective

Charles and Ray Eames conceptualized a multi-media

expression of reality in new avant-garde film. The

presentation for an American exhibition in Moscow

dual logic of simple opposites such as in/out or up /

in the summer of 1959. On seven massive monitors,

down doesn’t count anymore. Triadic logic is what

seven different films ran at the same time documenting

counts: in/out/maybe. The new synaesthetic cinema

the American lifestyle of the 1950s: A Day in the Life of

corresponds to this type of logic. Cinematic examples

the United States.

are Brakhage’s Dog Star Man, Warhol’s Blue Movie,

From the mid-1960s onward, this new movement in art has been called expanded art or, when referring to film, expanded cinema. Besides these, expressions

Carolee Schneemann’s Fuses und Michael Snow’s Wavelength (1967) (Youngblood 1970, 82–83). Despite all Youngblood’s efforts, the term synaes-

such as “intermedial”, “transmedial” or “mixed-media”

thetic cinema could not prevail against the term ex-

emerged. Such terms as “polymediality” or “poly-

panded cinema. Maybe because Youngblood focused

esthetic” in which the prefix “poly” didn’t mean “many”

on a single medium with his term synaesthetic cinema:

in terms of quantity but rather in terms of multiple

film. Those artists and producers who understood

qualities were already in use. The expression “poly-

their work as expanded cinema no longer understood

artist” was coined in 1968 by the producer Richard

the film screen as a boundary in their artistic work.

71

Jeffrey Shaw and Peter Weibel: From Expanded Cinema to Digital Synesthesia For decades, the artists Jeffrey Shaw and Peter

(Fig. 7). From a higher, more privileged standpoint the

Weibel have studied the domain of perception with

viewer can navigate through an artificial image land-

the help of new media. It therefore seems obvious to

scape via joystick. The digital images can be zoomed

view these artists’ work from an art-historical stance

into to the level of pixel resolution, and verses by Dirk

and in connection with digital synesthesia.

Groeneveld can be heard. The fixed standpoint is reminiscent of a classical perspective in Renaissance

tion of media artists to have worked on expanding the

painting. Instead of perceiving illusory depth percep-

boundaries of perception with expanded cinema since

tion, the viewer can zoom into the individual picture

the mid-1960s, and are associated with it to this day.

points or look at their pixel construction. With legendary works such as Legible City

the experimentation with new technologies; in their

(1988 –  91) or The Golden Calf (1994) Shaw created a

artistic methods and orientation very different

milestone in media art as well as archetypical digital

approaches are noticeable. In Shaw we can see an

synesthetic installations (Fig. 8, 9). Among them are

artistic approach using participatory and narrative

his work in 3D on a 360° projection surface, such

aspects in his work (Fig. 3).

as T Visionarium I and II (2003–2010). The extent to

Weibel, on the other hand, pulls his artistic work

which he is interested in experiments in new narrative

together with his theoretical work — as an epistemic

potential can be seen in his 2014 work Infinite Line

tool broaching the topics of philosophy, sociology,

(in collaboration with Sarah Kenderdine and Edwin

evolution or mathematics (Fig. 4 a-b).

Nadason Thumboo). In a 360° theater, the viewer can

For Shaw the moving image and poetry are central themes. Because the works are conceptualized inactively — the viewer should participate in the

call up 27 poems by Thumboo and create their own personal poem (Fig. 10 a-b). The media artist Weibel developed a series of com-

work — the artist provides an open structure. Similarly

munications concepts in the 1960s that dealt directly

to the artist Graham Stevens (1965), Shaw and Tjebbe

with the expansion of sensory perception. Examples

van Tijen created pneumatic sculptures in urban

are Distanz Sinne (Distance Senses) (1968), the

environments in order to intensify several senses at

Pillen-Project (Pill Project) film (1968) or Information

the same time (Fig 5). “Just as I love the friction

Unit (1967), a razor with multifunctional capabilities

between the real and the virtual, I also love the

such as radio, camera and tv (Fig. 11, 12, 13). Weibel

friction that ensues between a work that is my own

writes on the Pill movie: “I think of films in the form

and that which the audience can claim: it has also

of a new sensory organ that affords me every desired

become partially theirs. I like this type of union with

sensory quality, delivers air to me, whether coming

the audience. Traditional art has so few possibilities

from lung breathing, jam breathing or the reverse,

allowing this to happen and to be experienced”

that does not shy away from discrepancies of sensory

(Shaw 1993, 332).

data, permits me to grasp foam when I see stones,

In the endless filmstrip Continuous Sound and

allows me to smell chocolate when I eat grasses, lets

Image Music (1966) Shaw constructed a chronolog-

me hear violins when I glue envelopes shut, that

ical image perception for the individual sequences

allows me to feel water when I reach for the electrical

of a film (Fig. 6). The movie was shown at numerous

outlet, etc.” (Weibel 1968).

expanded art performances. Between 1985 and 1995 he created the interactive audio visual installation The Narrative Landscape

THEORIES ON DIGITAL SYNESTHESIA

Romana K. Schuler

A noteworthy similarity between the two of them is

Concepts which at that time were considered “too utopian” have become realizable today. For example Die Brille als Welt (Spectacles as World) (1967):

72

Digital Synesthesia: Reminiscing about Science and Art

Shaw and Weibel both belong to the first genera-

The spectacles are comprised of micro-computers in order to program an individual selection of images of the world. Weibel essentially anticipated Zuckerberg’s actual ideas of virtual reality. Weibel understands the act of seeing as an active experience that is not confined to the eyes. “The senses give us models of our surroundings, abstractly expressed theories, hypotheses” (Weibel, 1976, 20). He refers here to the book The Nature of Explanation (1952) written in 1943 by the perception researcher Kenneth Craik (Fig. 14). According to this work, intelligent interfaces of modern image technology should find direct access to the human body, and with that create total independence from space and time, additionally turning off the manipulable interface between image (information) and viewer. Eyes, mouth, ears and limbs, the biological senses interpret nature for us; they are similar to peripheral devices. When they are replaced by tools — by artificial peripheral devices like machines and computers — then a new interpretation of nature occurs. With the help of the term digital synesthesia (or also digital peripherals) the summing-up of new art historical concepts with narrative elements (Shaw) and epistemic ideas (Weibel) takes place. That means for current art history and art theory, the concept of digital synesthesia is as necessary for a no-longernegatable genre as is the question of the origins of digital art.

73

Digital Synesthesia: Reminiscing about Science and Art

THEORIES ON DIGITAL SYNESTHESIA Romana K. Schuler

Fig. 3

Fig. 5

74

Fig. 4a

Fig. 3 Jeffrey Shaw, Neil Brown, Dennis Del Favero, Matthew McGinity, Peter Weibel, T_Visionarium II, 2008 archive Jeffrey Shaw

Fig. 4a, 4b Peter Weibel, Life in the 20th Century | 222 Million Murders, 2011 Photo: Jens Barth Fig. 4b Fig. 5 Jeffrey Shaw, Tjebbe van Tijen, Sean Wellesley-Miller, Alpevent, Bern 1968 archive Jeffrey Shaw

75

Digital Synesthesia: Reminiscing about Science and Art

THEORIES ON DIGITAL SYNESTHESIA Romana K. Schuler

Fig. 6

Fig. 7

76

Fig. 8

Fig. 6 Jeffrey Shaw,Tjebbe van Tijen, Willem Breuhen (sound), Continous Sound and Image Music, 1966 archive Jeffrey Shaw

Fig. 7 Jeffrey Shaw, Dirk Groeneveld, The Narrative Landscape, 1985/1995 Photo: Oscar van Alphen

Fig. 8 Jeffrey Shaw, Legible City, 1988 – 91 Photo: Marco Caselli

Fig. 9

Fig. 9 Jeffrey Shaw, The Golden Calf (1994) archive Jeffrey Shaw

77

Fig. 10a, 10b Jeffrey Shaw, Sarah Kenderdine, Edwin Nadason Thumboo, The infinite Line, 2014 (archive Jeffrey Shaw)

Fig. 11 Peter Weibel, Distanzsinne, 1968 (archive Peter Weibel)

Romana K. Schuler

Fig. 12 Peter Weibel, Pillenfilm, 1968 (archive Peter Weibel)

Fig. 13 Peter Weibel, Information Unit, 1967 (archive Peter Weibel)

Fig. 14 Peter Weibel, The Nature of Explanation, 1973 (archive Peter Weibel)

Digital Synesthesia: Reminiscing about Science and Art

Fig. 10a

Fig. 10b

THEORIES ON DIGITAL SYNESTHESIA

78

Fig. 11

Fig. 12

Fig. 13

Fig. 14

79

Argelander, Annelies, 1927, Das Farbenhören und der synästhetische Faktor der Wahrnehmung, Jena: Gustav Fischer. Barash, Shabtai, 2006, “The mindbody problem: neurophysiology of looking and seeing”, in Life Science 2006, Rehovot: Weizmann Institute of Science. Behrens, Roger, 2014, “Digitale Frist. Computer. Pop (Beta Vision)”, in Bug Report Digital war besser, Mainz: Testcard 24, 42– 45. Brunswik, Egon, 1922, “Prinzipienfragen der Gestalttheorie”, in Beiträge zur Problemgeschichte der Psychologie. Festschrift zu Karl Bühler’s 50. Geburtstag, Jena: Verlag Gustav Fischer, 95 – 97. Chladni, Ernst F. Friedrich, 1787, Entdeckung über die Theorie des Klanges, Leipzig: Weidmanns Erben und Reich.

Knapp, Ernst, 1978, Grundlinien einer Philosophie der Technik [1877], Düsseldorf: Stern-Verlag, Janssen & Co. Kostelanetz, Richard, 1980, The Theater of Mixed-Means. An Introduction to Happenings, Kinetic Environments and Other Mixed-Means Presentations [1968], New York: RK Editions. Kuo Michelle, 2008, Special Effects. Michelle Kuo speaks with Micheal Challahan about USCO, (www.questia.com/magazine). Mach, Ernst, 1861, “Über das Sehen von Lagen und Winkeln durch die Bewegung des Auges. Ein Beitrag zur Psychophysik”, in Wiener Sitzungsberichte, Classe II, vol. 43, Vienna, 215 –224. Mach, Ernst, 1865, “Über die Wirkung der räumlichen Vertheilung des Lichtreizes auf die Netzhaut“, in Wiener Sitzungsberichte, Abt. II, vol. 52, Vienna 303 –322.

Purkinje, Jan, 1979, “Abhandlung über die physiologische Untersuchung des Sehorgans und des Hautsystemes“ [dissertation Breslau 1823], reprint in Acta Historica Leopoldina, nr 11, Halle/Saale: Deutsche Akademie der Naturforscher Leopoldina.

Stumpf, Carl, 1955, Über den Psychologischen Ursprung der Raumvorstellung [18, Amsterdam: E. J. Bonset.

Rösch, Gabriele, 2007, “Albert Wellek: Synästhesie als kulturbildendes Phänomen“, in Ausdruck Ausstrahlung Aura, Synästhesien der Beseelung im Medienzeitalter, K. Clausberg, E. Bisanz, C. Weiller eds., Bad Honnef: Hippocampus Verlag, 13 –28.

Weibel Peter, 1976, “Kommentare zur Natur der Technik”, in Konrad Balder Schäuffelen, sprache ist fuer wahr ein koerper, München: Städtische Galerie im Lenbachhaus, 19 –24.

Schuler, Romana Karla, 2014, “The Experiments of Perception in Science and Art by Ernst Mach, Dan Graham and Peter Weibel” in Art Theory as Visual Epistemology, Harald Klinke ed., Newcastle upton Tyne: Cambrigde Scholars Publishing, 125 –144. Schuler, Romana Karla, 2016, Seeing Motion. A History of Visual Perception in Art and Science, Berlin, Boston: De Gruyter. Shaw, Jeffrey, 1993, “Reisen in virtuelle Realitäten”, in F. Rötzer, P. Weibel, eds., Cyberspace. Zum medialen Gesamtkunstwerk, Munich: Boer Verlag.

Weibel, Peter, 1968, “Pillen-FilmProjekt“, in Supervisuell nr 2, Zürich.

Wiener, Nobert, 1952, Mensch und Mensch-Maschine, Berlin, Frankfurt a. Main: Alfred Metzner Verlag. Youngblood, Gene, 1970, Expanded Cinema, New York: E. P. Dutton & Co., Inc.

Digital Synesthesia: Reminiscing about Science and Art

Craik, Kenneth, 1952, The Nature of Explanation [1943], Cambridge: University Press

Fechner, G. Theodor, 1860, Elemente der Psychophysik, Leipzig: Breitkopf & Härtel.

Romana K. Schuler

References

THEORIES ON DIGITAL SYNESTHESIA

80

The Incidental Synesthete

Jeffrey Shaw / Sarah Kenderdine

  Keywords   multi-sensory,   prosthetic,   kinesthetic, embodiment,   virtuality,   panoramic,   embodiment,   place,   nature,   manufacture,   ataxia  

Sensory experiences are at the forefront of cultural

Embodiment offers us a strategy to revive the “synes-

analysis today, overturning linguistic and textual

thetic project” in the trajectory of the post-human and

analysis and supporting both experiential and phe-

hyper-virtual, and allow it to cohere with the promise

nomenological inquiry. Embodiment theories attempt

of “the art to come.” “Synesthetic art historically refers

to understand the mind as a set of physical processes

to multi-sensory experiments in the genres of visual

derived throughout the brain and body of a human,

music, music visualization, audiovisual art, abstract

which ultimately serve his or her action in the physical

film, and intermedia. Distinct from neuroscience, the

world. Embodiment is multisensory and results from

concept of synesthesia in the arts is regarded as the

effects of visual, auditory, tactile, olfactory, and

simultaneous perception of multiple stimuli in one

gustatory cues; it is an immersive entanglement

gestalt experience” (Wikipedia). These “multi-sensory”

through and with context and environment. The body

experiments were a hallmark of art practices that

carries time into the experience of place, and land-

emerged in the 50s and 60s, and were an ineluctable

scape and past and present fold in upon each other.

inspiration for artists who were looking for new

81

avenues of expression and experience. For example,

inflatable “eventstructures” (Waterwalk 1969) and

earlier endeavors in “visual music” such as Alexander

“expanded cinema situations” (Corpocinema 1967)

Scriabin’s color organ Chromola were given new life

which were manifestly directed at incorporating the

in the liquid light shows that surfaced in the early

viewer’s body as being integral to the expression and

1960s as an accompaniment to electronic music and

operation of the artwork. and Sarah Kenderdine’s recent joint practice, albeit

music. Furthermore, the artists Mark Boyle and Joan

as a concomitant to research that focuses on strat-

Hills took the legacy of Piero Manzoni’s seminal

egies of immersive interactive visualization. In this

explorations of the conjunction between art

respect we can single out one particular installa-

production and human production (Merda d’artista,

tion, Place-HAMPI (2006), to elucidate the dynamic

Fiato d’artista) to directly infuse these liquid light

“multi-sensory” conjunction that emerges from the

shows with their own bodily fluids.

conjunction of 3D panoramic visualization with the

In the installation Black Airground (1968) at the

performative engagement of an audience, which is

Oxford Museum of Modern Art, Jeffrey Shaw used

actualized by the unique design of a technological

a first generation sound reactive light controller to

apparatus. Assuming the reader is already familiar with

modulate three 1000-watt light bulbs at various

this installation (http://www.place-hampi.museum/)

frequencies of sound input, so that the illumination

we focus on a visitor survey done by Anita Kocsis

of the exhibition space was entirely dependent on

during its exhibition at the Martin Gropius-Bau in

vocalization from the visitors. This was an expression

Berlin in 2008. In this particular questionnaire, the

of a further and more fundamental development of

visitors were offered a simple outline drawing of

the contemporary “multi-sensory” experience

a human figure and asked to put marks on it in

whereby “interactivity” and the engagement of the

response to the question “where on your body did

viewer via various technological prostheses created a

you experience this work?”

dynamic conjunction of between the artwork and the

Figs. 1–6 show a selection of the responses that

bodily action of the viewer. A seminal work in this

were received. Fig. 7 is a composite drawing of all

respect was Julio Le Parc’s Glasses for a Different

visitor responses, which substantiate the notion

Vision (1965), which proposed the optical prosthesis

of embodiment as being at the heart of the multi-

as an interactive apparatus to re-view the world, and

sensory experiment.

thereby prefigures the now-emerging ubiquity of VR goggles such as the Oculus Rift and Google Glass.

In his book Body and Image: Explorations in Land-

And even more so than “virtual reality”, it is “augment-

scape Phenomenology (2008), Christopher Tilley

ed reality” (the audio and/or visual conjunction of real

contrasts iconographic approaches to the study of

and virtual elements) that signals one of the most

representation with a perspective based on kinesthetic

compulsive and eloquent expressions of “the simulta-

enquiry. “Iconographic approaches … grant the

neous perception of multiple stimuli in one gestalt

primacy to the human mind as a producer of the

experience” in our time. Jeffrey Shaw explored the

meaning of the images through sensory perception.

generative capacity of AR in numerous artworks

It is the mind that responds in a disembodied way.

including Virtual Sculpture (1981), Inventer La Terre

Kinesthetic approaches, by contrast, stress the role

(1986), and The Golden Calf (1994), and underlying

of the carnal human body. The general claim is that

each of these works was the principle of the viewer’s

the manner in which we perceive, and therefore

embodied engagement as a necessary precondition

relate to visual imagery, is fundamentally related to

for a synesthetic conjunction of perception and

the kinds of bodies we have. The body both limits

experience. That precondition also drove the

and constrains and enables us to perceive and react

formulation of his earlier audience participation

to imagery in specific embodied ways.”

THEORIES ON DIGITAL SYNESTHESIA

82

Jeffrey Shaw / Sarah Kenderdine

adapted for performances of rock or psychedelic

This precondition remains central to Jeffrey Shaw

The Incidental Synesthete

avant-garde theater performances and were later

One could say that IN_SIDE VIEW is an allegory for

conflagration, form and dissolution, purpose and

the multi-sensory encounter between the body

affliction, past and present, biology and machine,

and the perceptual imaginary, where the imaginary

the gratuitous and the necessary, between survival

itself embodies a further encounter between the

and ataxia. Nature-society is seen as a complex

natural and made world. In this respect it contributes

entanglement of people-things, here played out

to the paradoxical connection between the concepts

in an embodied theater of hybrid realities induced

of nature and culture proposed by Vilem Flusser

through immersive display systems. IN_SIDE VIEW

(Natural:Mind, 1967), by setting up an assembly of

concedes with a pataphysical riposte to these

confrontations and concurrences – amongst others

technological affordances that now constitute con-

between tongue and apparatus, between mouth and

temporaneity’s diametric spaces of phenomeno-

eye, between stone and vegetation, steel and

logical encounter.

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Figs. 1–6 These are a selection of the responses that were received.

83

Jeffrey Shaw / Sarah Kenderdine The Incidental Synesthete

Fig. 7 Co-Designing New Media Spaces, Anita Kocsis, 2007

THEORIES ON DIGITAL SYNESTHESIA

84

The Interaction between Seeing and Moving: Eye Movements as a Window to the Mind ? Miriam Spering

  Keywords   visual perception, eye movements, brain, cognition, visual neuroscience, active vision, perception-action dissociation, awareness

Seeing and perceiving the visual world is an active

Montagnini, 2011). Smooth pursuit eye movements

and often multimodal process that involves orienting

are finely tuned to the visual object’s properties

eyes, head and body towards an object of interest.

of motion, direction and speed, and help to extract

It is also a highly dynamic process during which the

visual motion information to enable more accurate

eyes continuously scan the visual environment to

and precise interactions with our environment

sample information. When we view the world around

(Spering et al. 2011). Furthermore, we have the

us, our eyes are always in motion. We move our gaze

ability to hold our gaze steadily on stationary objects

to fixate on objects of interest using fast eye move-

of interest using fixational eye movements, and to

ments called saccades. These small, highly accurate

compensate for movements of our head and body

movements occur at high speed and frequency – up

using the vestibular-ocular reflex. Failure to stabilize

to three times per second, more frequently than a

gaze or to move gaze to objects of interest results

heartbeat – during activities such as reading or scene

in motion blur, loss of visual acuity and impaired

viewing that require the perception of fine spatial

fine spatial vision. It thus seems that the link between

detail (Carpenter, 1988; Kowler, 2011). We also track

eye movements and visual perception is tight: we

objects with our eyes to keep them close to the

move our eyes in order to enhance and optimize

fovea – the area in the eyes producing the highest

the perception of visual objects and events in our

visual acuity – using a type of eye movement

environment, a process sometimes referred to as

termed smooth pursuit (Lisberger, 2010; Spering &

active vision (Findlay & Gilchrist, 2008).

85

regular and reproducible, and that a viewer’s eyes fol-

by many research disciplines, ranging from develop-

low a similar path when viewing an image repeatedly,

mental and cognitive psychology to computer science

reflecting how objects or images are remembered

and art history, to measure visual perception, object

and recognized (Noton & Stark, 1971). The discovery

categorization, recognition, and many other mental

that cognitive processes influence eye movements

processes. Eye movements are also utilized to design

was first made by Russian physicist and vision scientist

and assess products and applications (e.g., in web-

Alfred Yarbus, sometimes considered the founder of

page design) or to evaluate performance (e.g., in a

modern eye movement research (Yarbus, 1967; Tatler

driving simulator or in sports). These studies and

et al. 2010). Yarbus made several key observations:

applications are based on the fundamental assump-

first, when viewing a painted scene, the initial

tion that eye movements reflect what we see, perceive,

fixation is preferentially directed at persons within

think or desire.

the scene, and more specifically, to the individuals’

A rich literature in experimental psychology indi-

faces (Buswell, 1935; Fletcher-Watson et al. 2008)

cates that eye movements reflect awareness – the

and their eyes (Kingstone, 2009). Second, during

ability to explicitly report something – and can

longer viewing, gaze follows a cyclical viewing

serve as a key to our understanding of visual and

pattern and repeatedly revisits the most important

cognitive processes. For example, voluntary eye

elements of a picture. Third, gaze patterns depend

movements such as saccades and smooth pursuit

on instruction – a finding we have reproduced here

are driven by visual information (Kowler, 2011;

(Fig. 1; see also Castelhano, Mack & Henderson,

Spering & Montagnini, 2011) and their dynamic

2009; Tatler et al. 2010). In more recent years, eye

properties sensitively reflect how our brain processes

movement responses have been formalized to

visual information (Lisberger, 2010). When we look

investigate cognitive processes during visual search,

at an object or image, our gaze and the focus of our

reading, and scene viewing (for reviews, see Rayner,

attention are quickly attracted to locations of high

1988; 2009). Research in vision sciences has focused

salience (Itti & Koch, 2001) – conspicuous areas in

on the role of eye movements for visual perception.

the image that stand out from their context because

Many have shown that eye movements enhance

of their visual properties. Fig. 1a shows a photo of the

different aspects of vision; for instance, the ability to

Vancouver north shore. When asked to simply look

perceive fine spatial detail or to extract information

at this image, a viewer’s gaze is attracted by salient

about an object’s motion, direction and speed

objects or locations, such as the black birds in front

(for reviews, see Kowler 2011; Schütz, Braun &

of the blue sky, the red ships surrounded by a blue-

Gegenfurtner, 2011; Spering & Montagnini 2011).

grey context and the mountain peaks (Fig. 1b).

Eye movements are also critical in guiding visual

Interestingly, gaze patterns may also reflect a specific

processes during everyday activities that involve hand

cognitive task. When asked how many birds were in

and body movements, such as making a cup of tea

the photo, our viewer systematically scans water and

or playing sports (for reviews, see Hayhoe & Ballard,

rocks, and their gaze lingers on an ambiguous area

2005; Land & Tatler, 2009).

to the far left, where a seagull can be seen (Fig. 1c). We produced these eye movement patterns by showing the image only once, for 10 seconds. However, previous studies indicate that eye movement patterns during object or scene viewing are highly

THEORIES ON DIGITAL SYNESTHESIA

86

The Interaction between Seeing and Moving: Eye Movements as a Window to the Mind?

Eye movements are used in many contexts and

Miriam Spering

Eye Movements as a Window to the Mind

When Eye Movements Reflect the Processing of Unaware Information Recently, however, the established view of there being

throw and catch (Fig. 3a); in another condition, the

a tight link between visual perception and eye move-

movie showed the actor merely pretending to throw

ments has been questioned and several studies have

and catch a ball by making appropriate hand move-

revealed situations in which eye movements reflect

ments and following the imaginary ball trajectory with

the processing of visual information that the viewer

head and eyes (Fig. 3b). Most observers perceived the

is not aware of (for a review, see Spering & Carrasco,

illusion and also claimed that their eyes followed the

2015). We developed an experimental paradigm in

ball. Interestingly, their gaze was not fooled by the

which one part of the visual scene is suppressed from

illusion: while they perceived the ball to have been

awareness and not perceived, using an established

thrown, their eyes mostly remained fixated on or

perceptual procedure called binocular rivalry flash

around the actor’s head.

suppression. During binocular rivalry, different images

Many other studies have reported quantitative

are presented to the two eyes using mechanical

differences between perception and eye movements,

devices such as a stereoscope. Instead of merging

in which eye movements are more sensitive than our

the two images into one coherent percept, viewer

conscious perception. For example, the eyes may

perception usually alternates between the two images

sensitively react to small differences in the speed of

(Blake & Logothetis, 2002), indicating that one of the

a moving target that go completely unnoticed

two images is selectively suppressed from awareness.

(Tavassoli & Ringach, 2010).

We showed that the suppressed image can nevertheless influence eye movements (Spering, Pomplun &

This series of examples (reviewed in Spering & Carrasco 2015) reveals that eye movements may be

Carrasco, 2011). Our observers were shown two

more sensitive than visual perception and can some-

images on a computer monitor, viewed through a

times even reflect visual information that we are not

stereoscope, which projected the image on the left

aware of. Eye movements may be earlier, faster

side of the screen to the left eye only, and the image

and more accurate predictors of the way we locate

on the right side to the right eye only (Fig. 2a). One

and track events in the visual world compared

image showed a pattern moving to the right, the

with perceptual reports. Eye movements may thus

other image showed a pattern moving downwards.

be more than just a window to the mind: they

Congruent with research findings in binocular rivalry,

may serve as sensitive indicators of unconscious

observers reported seeing only one of the two images

visual processing.

at a time, either in a rightward or downward motion (example of downward shown in Fig. 2b). Remarkably, the eyes responded sensitively to the unperceived information as if it was completely visible, and moved diagonally down and right (Fig. 2c), to a motion direction that was almost never perceived. This study shows that we can perceive one thing while our eyes actively look at or track another object or part of a visual scene. Another impressive example comes from a study by Kuhn and Land (2006), in which observers were asked to view movies of an actor throwing and catching a ball. In one condition, the movie clip showed an actual

87

How Does the Brain Control Eye Movements without Awareness? One of the key questions is how these findings

than perception, can provide further insight into

can be reconciled with decades of research showing

brain function. We recently proposed that different

tight links between perception and eye movement.

brain pathways may mediate situations in which

The idea that perception and movement may be

eye movements reflect awareness and closely

processed differently in the brain is well-established

match perception versus situations in which eye

in the neurosciences: the dual-pathway model

movements reflect visual processing without

(Goodale & Milner, 1992; 2013) proposes largely

awareness (Fig. 4).

perception and vision-for-action, which may control

physiological, neuroimaging and clinical studies

movements without awareness (Milner, 2012;

now indicates the involvement of a subcortical

Goodale & Milner, 2013). Our novel findings,

pathway (analogous to the one shown in blue in

indicating that eye movements are more sensitive

Fig. 4) in unconscious visual processing.

Conclusions Visual perception is an active process that relies on

and more sensitive to visual information that we do

fast and accurate eye movements. The tight per-

not consciously perceive. Such perception-movement

ception-movement link can be utilized to infer the

dissociations may be adaptive responses to different

workings of the mind by evaluating the spatiotemporal

task requirements and may allow insight into visual

dynamics of eye movements. However in some situ-

processing without awareness and its underlying

ations eye movements may be faster, more accurate

brain mechanisms.

Fig. 1a

THEORIES ON DIGITAL SYNESTHESIA

Fig. 1b

Fig. 1c

88

Miriam Spering

Converging evidence from behavioral, neuro-

The Interaction between Seeing and Moving: Eye Movements as a Window to the Mind?

independent brain processing for vision-for-

Fig. 1 (a) Vancouver north shore mountains. (b) One viewer’s gaze pattern during 10 seconds of free viewing; the size of the red circles indicates fixation duration; red lines indicate saccades. (c) Gaze pattern when the same viewer counted the number of birds shown in the photo.

Fig. 2 (a) Experimental set-up showing two visual images on a computer monitor, viewed through a stereoscope (not shown) so that each image is seen by one eye only; in this example, the left eye viewed the rightward moving image, the right eye viewed the downward moving image. (b) Perception follows one motion direction exclusively. The red arrow represents mean perceived motion direction across eight observers. (c) Eye movements track the average motion direction of both images (pattern motion) – a direction that is perceived in less than 1 % of all trials. Adapted from Spering and Carrasco (2015).

Fig. 3 (a) A ball is thrown up in the air and caught again; observers’ gaze (red circle marks center of fixation) follows the ball. (b) Actor pretends to throw and catch the ball; observers’ gaze (red circle) is centered on or around the actor’s head. Images and schematic data points created following Kuhn and Land (2006).

Fig. 4 Schematic brain pathways for perception and eye movements in a human brain. The red pathway mediates visual information for perception (retinogeniculate pathway); the blue pathway mediates visual information for eye movements in the absence of awareness. This fast, retinocollicular pathway directly connects the eyes with brainstem structures and bypasses some of the structures involved in the retinogeniculate pathway. Adapted from Spering and Carrasco (2015).

Fig. 2a

Fig. 2b

Fig. 3a

Fig. 2c

Fig. 3b

Fig. 4

89

Research partly funded by a Natural Sciences and Engineering Research Council Canada (NSERC) Discovery Grant and the Peter Wall Institute for Advanced Studies at UBC.

References Blake, R. & Logothetis, N. (2002). Visual competition. Nature Reviews Neuroscience, 3, 1–11. Buswell, G. T. (1935). How people look at pictures: a study of the psychology of perception in art. Chicago, IL: University of Chicago Press. Carpenter, R. H. S. (1988). Movements of the eyes. London, UK: Pion Publishers. Castelhano, M. S., Mack, M. L., & Henderson, J. M. (2009). Viewing task influences eye movement control during active scene perception. Journal of Vision, 9(3):6, 1–15.

Goodale, M. A. & Milner, A. D. (1992). Separate visual pathways for perception and action. Trends in Neurosciences, 15, 20–25. Goodale, M. A. & Milner, A. D. (2013). Sight unseen: an exploration of conscious and unconscious vision. Oxford, UK: Oxford University Press. Hayhoe, M. & Ballard, D. (2005). Eye movements in natural behavior. Trends in Cognitive Sciences, 9, 188–194. Itti, L. & Koch, C. (2001). Computational modelling of visual attention. Nature Reviews Neuroscience, 2, 1–11. Kingstone, A. (2009). Taking a real look at social attention. Current Opinion in Neurobiology, 19, 52–56. Kowler, E. (2011). Eye movements: the past 25 years. Vision Research, 51, 1457–1483. Kuhn, G. & Land, M. F. (2006). There’s more to magic than meets the eye. Current Biology, 16, R950–R951. Land, M. F. & Tatler, B. W. (2009). Looking and acting: vision and eye

movements in natural behaviour. Oxford, UK: Oxford University Press.

pursuit eye movements: A review. Vision Research, 51, 836–852.

Lisberger, S. G. (2010). Visual guidance of smooth-pursuit eye movements: sensation, action, and what happens in between. Neuron, 66, 477–491.

Spering, M., Schütz, A. C., Braun, D. I., & Gegenfurtner, K. R. (2011). Keep your eyes on the ball: smooth pursuit eye movements enhance prediction of visual motion. Journal of Neurophysiology, 105, 1756–1767.

Milner, A.D. (2012). Is visual processing in the dorsal stream accessible to consciousness? Proceedings of the Royal Society B, 279, 2289–2298. Noton, D. & Stark, L. (1971). Scanpaths in eye movements during pattern perception. Science, 171, 308–311. Rayner, K. (1988). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372–422. Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology, 62, 1457–1506. Schütz, A. C., Braun, D. I., & Gegenfurtner, K. R. (2011). Eye movements and perception: A selective review. Journal of Vision, 11(5):9, 1–30.

Spering, M. & Carrasco, M. (2015). Acting without seeing: eye movements reveal visual processing without awareness. Trends in Neurosciences, 38, 189–258. Tatler, B. W., Wade, N. J., Kwan, H., Findlay, J. M., & Velichkovsky, B. M. (2010). Yarbus, eye movements, and vision. Iperception, 1, 7–27. Tavassoli, A. & Ringach, D. (2010). When your eyes see more than you do. Current Biology, 20, R83–R94. Yarbus, A. L. (1967). Eye movements and vision. New York, NY: Plenum Press.

Spering, M. & Montagnini, A. (2011). Do we track what we see? Common versus independent processing for motion perception and smooth

The Interaction between Seeing and Moving: Eye Movements as a Window to the Mind?

Findlay, J. M. & Gilchrist, I. D. (2008). Active vision: the psychology of looking and seeing. Oxford, UK: Oxford University Press.

Fletcher-Watson, S., Findlay, J. M., Leekam, S. R., & Benson, V. (2008). Rapid detection of person information in a naturalistic scene. Perception, 37, 571–583.

Miriam Spering

Acknowledgements

THEORIES ON DIGITAL SYNESTHESIA

90

Digital Synesthesia

Peter Weibel

  Keywords   computability, correlation, data, database, entropy, information,   music,   noise, randomness, undecidability

Analog synesthesia describes the processing of

functions. Evidently the visual cortex of the brain can

sensory data in the human brain. Analog synesthesia

also process auditory stimuli, likewise the area of

is the effect of an interior experience of a subject.

the cortex responsible for acoustic stimuli can also

The scientific explanation at the moment is the

process visual stimuli.

following: 1. A human being has several doors of perception

4. Due to this neuroplasticity of the brain, a cortical remapping can take place. As a consequence visual

like the eyes, the ears, etc. Through these sensory

stimuli can produce auditory stimuli and acoustic

organs light waves or sound waves, so-called

stimuli can produce visual stimuli.

stimuli, enter the brain. 2. The brain consists of different areas in which

These four steps describe the classical synesthetic experience, which around 1900 evoked great interest

these different sensory stimuli are processed.

and produced a lot of relevant art works, from

According to the classical location theory, different

Alexander Scriabin to Arnold Schönberg, from Wassily

areas of the brain are in charge of different

Kandinsky to Mikalojus Konstantinas Čiurlionis, i.e.,

sensory stimuli.

from synesthetic musicians to synesthetic painters.

3. But observation and experience show that neighboring areas of the brain can adopt different

Digital synesthesia is not a subjective experience, but an objective observable phenomenon. It does

91

not happen in the human brain and it has nothing to

repeated both in the same, and in several Papers, I

do with the processing of human sensory stimuli.

found that the Observations agreed well enough with

It happens in the physical world according to the laws

one another, and that the Rectilinear Sides MG and FA

of nature. Digital synesthesia does not connect and

were by the said cross Lines divided after the manner

intertwine sensory modalities, but operates within

of a Musical Chord. Let GM be produced to X, that MX

the physical properties of the world.

may be equal to GM, and conceive GX, λX, ιX, ηX, εX,

γX, αX, MX, to be in proportion to one another, as

At the beginning of digital synesthesia is Isaac Newton‘s Opticks, or a Treatise of the Reflexions,

the Numbers, 1, 8/9, 5/6, 3/4, 2/3, 3/5, 9/16, 1/2, and

Refractions, Inflexions and Colours of Light (1704). In

so to represent the Chords of the Key, and of a Tone,

his book, Newton summarizes the conclusions from

a third Minor, a fourth, a fifth, a sixth Major, a seventh and an eighth above that Key: And the Intervals Mα, αγ,

his discovery that white light consists of rays that

γε, εη, ηι, ιλ, and λG, will be the Spaces which the

refract differently and correspond to the color spectrum. Newton invented the art of systematic

several Colours (red, orange, yellow, green, blue,

experimentation as to be able to develop a testable

indigo, violet) take up.” 4 So he was the first to provide

theory. He therefore starts his book on optics with

a mathematical theory on the relationship between

the following sentence: “My Design in this Book is

colors and sounds.

not to explain the Properties of Light by Hypotheses,

This is precisely the origin of digital synesthesia:

but to propose and prove them by Reason and

The relationship between visual and acoustic

Experiments.”  We can hear Roger Bacon‘s (Opus

phenomena in terms of numbers and digits. As we

majus, 1266) words echoing here. The British

know, the Greeks had already described music as a

Franciscan monk and philosopher Roger Bacon

form of mathematics. But it was Newton who was also

1

able to describe color theory as a form of mathematics.

Newton was one of the first to see a parallelism between the world of light and the world of sound. Newton not only investigated optical laws

When music and light are just two different forms of mathematics then it is just consequent to link them objectively in a form of coefficient theory. Hermann von Helmholtz further expanded on

but also saw how these were related to the laws

the mathematical theory of acoustic and optical

of acoustics: “For as Sound in a Bell or musical String,

stimuli in his physiology of hearing and seeing, Die

or other sounding Body, is nothing but a trembling

Lehre von den Tonempfindungen als physiologische

Motion, and in the Air nothing but that Motion

Grundlage für die Theorie der Musik (1863). 5 With

propagated from the Object, and in the Sensorium ’tis

his mathematical studies, Helmholtz pinpointed new

a Sense of that Motion under the Form of Sound; so

links between physics, physiology, and aesthetics,

Colours in the Object are nothing but a Disposition

as Gustav Theodor Fechner did with his theory of

to reflect this or that sort of Rays more copiously than

psychophysics (Elemente der Psychophysik, 1860). 6

the rest; in the Rays they are nothing but their Disposi-

After Newton, James Clerk Maxwell was the

tions to propagate this or that Motion into the

first to provide a mathematical representation of

Sensorium, and in the Sensorium they are Sensa-

electromagnetic waves, the carrier of light (color)

tions of those Motions under the Forms of Colours.” 3

and sound. Maxwell‘s equations are a set of partial

Newton described sound and light as propagation of

differential equations that form the foundation of

motion. He described music and color as scientific

classical electrodynamics, classical optics, and

objects. Digital synesthesia is a scientific technical

electric circuits. Maxwell‘s equations describe how

operation. The most important discovery was that

electric and magnetic fields are generated and altered

Newton also devised a kind of coefficient theory

by each other.

between sounds and colors that he defines numerically: “And this Operation being divers times

THEORIES ON DIGITAL SYNESTHESIA

Peter Weibel

knowledge, namely, by reasoning and experience.” 

2

Between 1861 and 1864 Maxwell devised an electromagnetic theory of light in which he first asserted the

92

Digital Synesthesia

wrote: “For there are two modes of acquiring

existence of electromagnetic waves and then equated

floats and convulses in darkness to a stark and powerful,

the speed of light with the speed of electromagnetic

highly synchronized soundtrack. The velocity of the

waves.  Heinrich Hertz, a student of Hermann von

moving images is ultra-fast, some hundreds of frames

Helmholtz, conducted between 1886 and 1888

per second, so that the work is a performance test for

thought experiments and later lab experiments in

the audio and visual devices, as well as a response test

Karlsruhe with sparks and vibrations, enabling him

for the audience’s perceptions, pushing it to its limits.

7

to identify electromagnetic waves and light waves.

As we can learn from this example digital

Radio, TV, radar, and cordless phones have become

synesthesia introduces a new concept of music and

the results of this wireless communication of sounds

image. Musical and visual systems are now positioned

and images by electromagnetic waves. Hertz is thus

between entropy and information, noise and order.

considered one of the originators of physical vibra-

In the area of digital synesthesia the artist operates

tion theory, which applies to both optic and acoustic

with algorithms, bits, mathematical precision, e.g., the

waves. The unit for the frequency at which these

logarithm of the number of possible options. The

occurrences are repeated per second in a periodic

classical correlation between optics and sound now

signal was named after Heinrich Hertz in 1930 and

has a new basis: The correlation between information

introduced in 1935 as Hz. Despite the definition, the

and entropy.

use of the unit is not restricted to periodic vibrations.

From Newton’s Experimentum Crucis 9, in which

Regularly repeated events, such as the frequency at

white light is shown to be the sum of all prismatic

which a computer creates backup copies of files or

colors, we can learn that noise may be the optimal

issues commands can also be measured in Hz

state of music, the summa maxima of all melodies.

(frequency). Modern processors have a frequency

Noise could be the white light, containing all colors

in the gigahertz (GHz) range, meaning a billion

and sounds. The experimentum crucis of digital

cycles a second.

synesthesia is therefore the following: Defining music

Many contemporary sound experiments are based

as entropy and information in digital systems marks

on these transversal and longitudinal vibrations. In

the beginning of a digital universe of sounds in

their collaborative research project, cyclo. id (2000),

which each instrument can emulate the sound of any

which focuses on the visualization of sound, the artists

other instrument and each object can simulate the

Ryoji Ikeda and Carsten Nicolai visually register each

sound of any other object.

wave form created by different frequencies. Approxi-

In the analog world there was a correlation

mately more than a thousand waveforms are depicted

between sensory organ and sensory stimuli. The eye

establishing a kind of virtual Hertzian library of sound.8

was the organ to process light waves, a very small

The artists developed a database of sounds, using

spectrum of the electromagnetic waves radiated by

equipment developed originally for phase correlation

the sun. In a certain sense the eye is therefore

in mastering vinyl records. The phase and amplitude

evolution’s answer to the sun. The ear was the organ

of stereo signals were graphically illustrated. In digital synesthesia the psychophysical parallels

to process sound waves, also a very small spectrum. Our natural sensory organs are very limited. They do

between acoustic and visual stimuli happen at the

not perceive e.g., the so-called radio waves, not to

purely physical, i.e., mathematical level, and are

mention the cosmic microwave background radiation,

arrived at by means of a computer-based scheme. As

the oldest light in the universe. Similarly, there is in

an example we can consider the work by Ryoji Ikeda.

the analog world a correlation between instrument

In his series test pattern (since 2008) he uses a system

and sound, e.g., a piano makes piano sound, and

that converts any type of data (text, sounds, photo-

between object and sound, e.g., each car creates a

graphs, and films) into barcode patterns and binary

typical sound.

patterns of zeros and ones in real time. Ikeda presents intense, flickering, black-and-white imagery that

In the digital world the tie between sensory organ and sensory stimuli, between instruments and sound,

93

objects and sound, is completely broken and inter-

produce any sound. A chair can sound like a piano,

rupted. The source of the sound and the sound itself

a piano can sound like a violin, a violin can sound like

have no one-to-one relationship anymore, because in

a drum, and a glass of wine can sound like wind. All

the digital age the computer turns the sound into data

these acoustic phenomena can also produce visual

and the computer decides which sensory modality

phenomena in real time. The visual phenomena can

is chosen. The computer can turn sound data into

be fed back to the sound sources and change the

light and visual data into sound. The computer has

sound. These sounds can be fed back to the visual

become the universal organ. A piano can deliver

sources and change the visuals. We can create cycles

typical piano sounds, the computer can process them

and circuits of sounds and light, of music and colors

and turn them into violin sounds and additionally

in infinite variations, which only rely on Relay and

create visual patterns. Digital synesthesia is locally

Switching Circuits (Claude E. Shannon, 1938), that is

bound neither to the brain nor to any instrument.

on electromagnetic waves.

The universal brain or instrument is the computer.

Digital synesthesia implies that music is a variable,

The human sensory organs are replaced by technical

not even bound 1:1 to an instrument or object. Like

man-made sensors. We can place a sensor on any

noisy radar data, acoustic data becomes a matter of statistics. Noise is part of any signal and what is noise and what is music always remains uncertain.

perception (eyes or ears) is decided by the computer.

Noise and Randomness, Information and Incomplete-

The computer can modulate the sound, can store

ness, Computability, and Undecidability form the

it, and can rhythmize it. Therefore any object can

parameters in the age of digital synesthesia.

1 Isaac Newton, Opticks: Or, a Treatise of the Reflexions, Refractions, Inflexions and Colours of Light, London, 1704, p. 1.

3 Isaac Newton, Opticks: Or, a Treatise of the Reflexions, Refractions, Inflexions and Colours of Light, London, 1704, p. 109.

2 Roger Bacon, Opus majus [1266], transl. by Robert Belle Burke. Vol. II, New York: Russell & Russell 1962, part 6, ch. 1, p. 583.

4 Ibid., p. 111.

6 Translated into English by Herbert Sidney Langfeld: Elements of Psychophysics [1860], 1912. First published in: Benjamin Rand (Ed.), The Classical Psychologists. Boston: Houghton Mifflin, pp. 562–572.

5 Translated into English by Alexander J. Ellis: On the Sensations of Tone as a Physiological Basis for the Theory of Music, London: Longmans, Green, and Co., 1875.

7 James Clerk Maxwell, “A Dynamical Theory of the Electromagnetic Field,” in: Philosophical Transactions of the Royal Society of London, 155, 1865, submitted in 1864, pp. 459–512.

Peter Weibel

object and any object can produce sound – but what happens to the sound before it arrives at our organs of

THEORIES ON DIGITAL SYNESTHESIA

94

9 Isaac Newton: “Newton to Oldenburg. 6 February 1671/2. From the Philosophical Transactions, 6 (1671/2), 3075–87”, in: H.W. Turnbull (ed.): The Correspondence of Isaac Newton. Vol. I: 1661–1675. Cambridge: University Press 1959, pp. 92–107, here p. 94.

Digital Synesthesia

8 See e.g., p. 36 in: Ryoji Ikeda and Carsten Nicolai, cyclo id. Berlin: Gestalten 2011.

Fig. 1

Fig. 1 Peter Weibel: The Origin of Noise – The Noise of the Origin, 3D-noise concert, world premiere at Donaufestival Krems Brûlée, 25.04.2013. ZKM | Karlsruhe 2013 ©: ZKM | Karlsruhe Photo: Moritz Büchner

Fig. 2 Peter Weibel: The Origin of Noise – The Noise of the Origin, 3D-performance opera at BEYOND Festival, Karlsruhe, 02.10.2013. ZKM | Karlsruhe 2015 ©: ZKM | Karlsruhe Still image: Christina Zartmann

Fig. 2

95

Peter Weibel

Fig. 3

Digital Synesthesia

Fig. 4

Fig. 5

THEORIES ON DIGITAL SYNESTHESIA

96

Fig. 3 Ryoji Ikeda: the planck universe [macro], 2015 Audio-visual wall projection, 3 video projectors, computer, loudspeaker (Exhibition view “Ryoji Ikeda: micro | macro” ZKM | Karlsruhe 2015) © ZKM | Karlsruhe Photo: Martin Wagenhan

Fig. 4 Test protocol by Heinrich Hertz, showing the measuring results of the development of electromagnetic waves. (General Photograph Collection of the Karlsruhe Institute of Technology (KIT), KIT-Archive)

Fig. 6

Fig. 5 Isaac Newton’s Experimentum Crucis from: “Newton to Oldenburg. 6 February 1671 /2. From the Philosophical Transactions, 6 (1671 /2), 3075 – 87”, in: H.W. Turnbull (ed.): The Correspondence of Isaac Newton. Vol. I: 1661 – 1675. Cambridge: University Press 1959, 92–107, here 106a

Fig. 6 The fields of a dipole antenna, from: Heinrich Hertz: “The Forces of Electric Oscillations, Treated According to Maxwell’s Theory” [1889], in: Id.: Electric Waves. Being Researches on the Propagation of Electric Action with Finite Velocity through Space. Translated by D.E. Jones, London: Macmillan, 1893, chapter 9: 137–159, here: 144 –145

Fig. 7 Experimental set-up: Transmitter and two coupled circuits. Photograph taken in Hertz’s laboratory of his Physics Institute at Karlsruhe, around 1887. (General Photograph Collection of the Karlsruhe Institute of Technology (KIT), KIT-Archive)

Fig. 7

97

This book chapter combines ten theor which have been written by neuro- an media and art theorists. The articles re between media aesthetics, synesthetic modalities, theories of embodiment, d issues in the context of DIGITAL SYNE of the international DIGITAL SYNESTH University of Applied Arts Vienna on A Peter Weibel

Fig. 8

Fig. 9 Digital Synesthesia

Fig. 8 Peter Weibel: The Origin of Noise – The Noise of the Origin, 3D-performance opera at BEYOND Festival, Karlsruhe, 02.10.2013. ZKM | Karlsruhe 2013 ©: ZKM | Karlsruhe Photo: ONUK

Fig. 9 Peter Weibel: Data Music, 2016 (Exhibition view AIL Vienna) Photo: Birgit and Peter Kainz

THEORIES ON DIGITAL SYNESTHESIA

98

retical texts on DIGITAL SYNESTHESIA nd cognitive scientists as well as by efer to topics regarding the relationship c perception, brain activities, sensory digital technologies, and philosophical STHESIA. They are also the proceedings HESIA symposium which was held at the April 4 and 5, 2016.

99

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA Part 2

This book chapter introduces 15 DIGIT been produced by 17 renowned media arts-based research process. The digit mental set-ups for the artistic research Texts, sketches and pictures provide in of the works in the research context o

THEORIES

100

THEORIES 102 – 105 Anke Eckardt

VERTICAL 2 106 – 109 Karl Heinz Jeron

Space Time 110 – 113   kondition pluriel

At Play 114 – 117   kondition pluriel

TAL SYNESTHESIA artworks which Diver have 118 – 123   Karen Lancel / Hermen Maat a artists in the course of a three-year E.E.G. KISS 124 – 125   A lan Kwan tal artworks have to be seen as experiThe Flying Umbrella Project h of cross- and multi-modal 126 perception. – 131  Marcello Mercado Bestiary for the Minds of the nformation about the artistic concepts 21 Century: Genomic Opera of DIGITAL SYNESTHESIA. 132 – 135   U lla Rauter st

Sound Calligraphy 136 – 139   R uth Schnell

facades 140 – 143   R uth Schnell

MotU #4–#6 144 – 147   R uth Schnell

Topography of Movement 148 – 149   J effrey Shaw/Sarah Kenderdine

IN_SIDE VIEW 150 – 153  David Strang / Vincent Van Uffelen

Transmission+Interference 154 – 157   Tamiko Thiel / Christoph Reiserer

I am Sound 158 – 162  Peter Weibel

Data Music

101

VERTICAL 2 Anke Eckardt

Anke Eckardt

Artistic media: Installation Primary sensory modalities: Kinesthetics (Balance, Visceroception), Audio, Vision 2016

VERTICAL 2

A

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

102

“How real is

B

A VERTICAL 2 (Sketch, Multisensory Installation by Anke Eckardt)

VERTICAL 2 is a continuation of a series of artworks by Anke Eckardt B VERTICAL 1: ! (Multisensory Installation by Anke Eckardt) A vertical line of six loudspeakers plays a downward sliding tone followed by a heavy sub-bass punch on the ground. The punch seems to trigger an “eruption” in a tank, which is filled with black liquid. The liquid erupts in the form of a round wave, which disintegrates and splashes in all directions. Because of the temporal and spatial conjunction of the two events, the “falling sound” and its “visual impact” right after, we interpret the scene as cause and effect. Synesthetic perception enables the physical experience of an intermodal illusion, realised by means of audio spatialization with multichannel loudspeakers and synchronized pneumatic technology. Photo: Holger Kist

on elements of architecture called SONIC SPACES: VERTICAL 2 (2016), GROUND (2014), WALL (BETWEEN | YOU | AND | ME 2012), VERTICAL 1 (“!” 2010). In this series, the artist explores the ambivalence of sensation. In VERTICAL 2 confusion, disinformation and communication are provoked by a small white cuboid standing in a large exhibition space. It can be entered by one person at a time. Contradictory, even paradoxical spatial information is produced via visual and audible stimuli. Low and small from the outside, the space gains unexpected height and thereby largeness when entered. The absence of consistency between the inner and outer perspective is furthermore deepened by another aspect to be experienced inside. The cuboid not only seems to open up vertically, but also downward below the feet. German philosopher Arthur Schopenhauer broached the issue of architecture as a complex staging system and cultural scientist Hartmut Böhme wrote about spatial atmospheres; not only the arrangement of solid construction material, but also acoustic perspectives and the designed behavior of light play one’s part at this juncture. Famous examples are e.g., St. Paul’s Cathedral in London (by architect Sir Christopher Wren,

103

Anke Eckardt

s real?”

VERTICAL 2

C

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

104

D

consecrated in 1708), which contains a Whispering Gallery, and Swiss C WALL: BETWEEN | YOU | AND | ME (Multisensory Installation by Anke Eckardt) BETWEEN | YOU | AND | ME is a wall of sound and light people can walk through. Four projectors and two lazers are used for creating precisely arranged beams of light. They form a visible frame that is filled with sound. The sonic architecture within the wall, textures of breaking glass, is realised by six extremely vectored hypersonic loudspeakers – a row of overlapping sonic beams and acoustic absorbers nearby.

D GROUND (Multisensory Installation by Anke Eckardt) The ground we stand on is in motion. 96 elements of concrete are moved by immense mechanical forces – motors, air bags, a compressor. The subtle motion can be heard, seen and felt – visitors might experience the loss of their visual reference points, rough sounds are produced just mechanically through friction between the concrete elements, there is an afterglow of a moving ground in the visitors physical memory after leaving the installation.

architect Peter Zumthor’s buildings, with regard to the staging of light or the acoustics of the skyscraper I.M. Pei, Bank of China, built in Hong Kong in 1990. The facade includes a large waterfall dividing into several smaller ones. Not only is surrounding traffic masked, in Chinese philosophy water is also considered to be a medium of immortality, now part of the spatial atmosphere of the Bank. Anke Eckardt precisely stages light and sound and their perceptional interplay into her installations. Inside the small cuboid of VERTICAL 2, the space seems to become an open vertical tunnel. Neuronal information about direction is given via reflected light, information about size via acoustic impulse responses. It might catch the visitor’s eye that the constructional elements of the cuboid are visible from the outside in a reductionist manner. Its building style lies open; there seem to be no secrets. All the more visitors are left alone with the ambivalence of their sensation. The realness of the outside world is considered to be independent of human observations. In this regard VERTICAL 2 appears to be an intentional aesthetic object that questions realness and enables visitors to synthesize virtuality. “In the way that swells of sensory information cause a constant flow of fluid modulations in the human body, Anke Eckardt’s projects function at the interstices of subtle (yet still acute) influence. Precisely interwoven she is using digital media like sound / ultrasound / infrasound and light as artistic material. Her pieces work to pick at the seams of sensorial determinism and reveal the slippage between modes of vibratory affect. In short, her installational projects become exemplars for a new kind of liquid agency; a future-facing state whose shores are lapped by the recent surge in augmented perception.” (Toby Heys)

105

Space Time Karl Heinz Jeron

Karl Heinz Jeron

Artistic media: Radio controlled clocks, custom electronics Primary sensory modalities: Kinesthetics, Chronoception, Audio 2016

Space Time

A

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

106

“I hear the buzz of

In general my artistic work is about reflection and perception. Space Time comprises the competition between two languages mediating complexity. One is the language of science. The other is the language of art, which has to understand science but doesn’t want to become it. The idea of what good science is or should be, the images of science and scientificity, are difficult to grasp but no less significant for it. With my project Space Time, A Studio shot development of 12 radio clock sound objects for Space Time installation

I am interested in the clock drift phenomena. Clock drift refers to several related phenomena where a clock does not run at the exact right speed compared to another clock. That is, after some time the clock “drifts apart” or gradually desynchronizes from the other clock. Everyday clocks such as wristwatches have finite precision. Eventually they require correction to remain accurate. The rate of drift depends on the clock’s quality, sometimes the stability of the power source, the ambient temperature, and other subtle environmental variables. Thus the same clock can have different drift rates at different occasions. Atomic clocks are very precise and have nearly no clock drift. Even the Earth’s rotation rate has more drift and variation in drift than an atomic clock. The principle behind the atomic clock has enabled scientists to define time in terms of a fundamental physical constant, the rate at which a type of particle oscillates. Measuring these events discretely (without skipping an event) is what gives an atomic clock nearly zero drift. Time signal stations synchronize their clocks to coordinated universal time (UTC), the international standard for timekeeping. No clock keeps coordinated universal time exactly because coordinated universal time is an average time, calculated with data collected from hundreds of atomic clocks located around the world. The accumulated time error of an analog radio-controlled clock will be measured between synchronizations. A sensor will be used that records the beat-rate of the watch’s pulses. Comprising several radio-controlled clocks, the Space Time installation will make this inaccuracy audible with clocks, electronic oscillators and speakers. A radio-controlled clock will be most accurate immediately after a successful synchronization and will become less accurate from that point forward until the next synchronization. To demonstrate this, the accumulated time error of an analog radio-controlled watch was measured between synchronizations. A sensor will be used that could record the beat rate of the watch’s pulses. Synchronization accuracy is limited by transmission system delay, path delay and synchronization errors.

107

Karl Heinz Jeron

of the clock drift.”

B

Space Time

C

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

108

Once the time synchronization signal leaves the transmitter and enters free space, it travels at the speed of light to the radio-controlled clock, approximately 0.000 003 336 seconds per kilometer. If you know the location of your radio-controlled clock and the location of the time signal station, you can calculate the distance between the transmitter and the receiver, and estimate the length of this delay. For example, a 3000 kilometer path would delay the time signal by about 10 milliseconds (0.01   s). This assumes, of course, that the signal travels along the ground and covers the shortest possible distance between the station and the radiocontrolled clock. With the low frequencies used by the time signal stations, a groundwave path can be assumed for short distances, perhaps up to 1500    km, but at longer distances the signal might bounce off the ionosphere (skywave) and take slightly longer to arrive. Even so, it is B Loudspeaker, radio clocks, oscillator control for sonification of time synchronizing deviation

safe to assume that the path delay will be less than 20 milliseconds (0.02 s) because the signal will rarely be usable at distances of more than 5000   km, and any additional delays introduced by skywave will be relatively small. There will be a measurable inaccuracy which will be

C Sensor, radio clock, DCF receiver (Installation 2 of 12 radio clock sound objects)

different with different brands and types of radio-controlled clocks. Space Time uses sonification to make time tangible. Sonification is the data-dependent generation of sound, if the transformation is systematic, objective and reproducible, so that it can be used as scientific method. A distinction between data and information is irrelevant with regard to the definition: information like, for instance, a message can always be represented numerically and thus be understood as data. Sonification refers to the technique and the process; the algorithm. Sonifications of the measured sensor data may be heard as music. According to the measured time inaccuracy, the sound, the color of the sound, and the volume of the sound are varied. The time inaccuracy adds a certain chance aspect to Space Time. Chance is the creative spark which excites artistic imagination and causes non-predictable but feasible material results. Thus, a sort of improvised piece of music is created from the simplest electronic parts. The sound installation refers to the concept of aleatory music, based on aspects of chance and improvisation as used by John Cage, Pierre Boulez, and Iannis Xenakis. Space Time depends on Charles Sanders Peirce’s notion of Tychism; rather than postulating the importance of determinism in cosmology and in socio-economic Darwinism.

109

At Play kondition pluriel (Martin Kusch / Marie-Claude Poulin)

kondition pluriel (Martin Kusch / Marie-Claude Poulin)

Artistic media: Responsive audio-visual installation with computer-tracking system Primary sensory modalities: Spatioception (Proprioception, Touch, Balance, Audio, Vision) 2016

At Play

A

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

110

“At the heart of our work is t the performativity existing in

B

At Play – a moment in the life of objects Might the empowerment of objects contribute to questioning our relationship to them, as well as the very meaning of their production? (David Bihanic)1 1 Empowering Users through Design: Interdisciplinary Studies and Combined ..., Ed. David Bihanic, 2015, Springer 2015

Every environment, its components and the relationships between them is enigmatic; a mystery to be penetrated, a space to comprehend. How should a place be read? Which decisions and actions should we carry out, and which relationships should we have? At Play is a responsive audio-visual environment that invites visitors to make stacking combinations with plastic storage bins to build the architectural landscape of the installation. Miniature versions of containers

111

kondition pluriel (Martin Kusch / Marie-Claude Poulin)

s the performativity of bodies, beings, machines and systems, but also in inanimate entities such as places and objects…”

At Play

C

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

112

that carry goods to world markets, these everyday objects, are playfully moved around by visitors to the exhibition like pieces of a puzzle. Resonance chambers, light sources and dynamic projection surfaces all at the same time, the boxes are integrated into a pattern recognition A, B, C “Enjeux” performance at Theatre La Chapelle, Montreal Photo: Bruno Colpron, © kondition pluriel, 2015

system that matches each sculptural composition with a particular visual and sonic atmosphere. The projected images and sound ambiances embed the visitors in fictional worlds in which they become the actors. Playing with building blocks that appear a priori analog, their actions reveal atomized sections of a world undergoing a perpetual mutation, calling upon their spatioception (perception of space). The grey material of the objects, fused with the lighting of the projected 3D imagery, comes to life through multiple trompe-l’œil nuances. The acoustic universe takes its source from the immediate environment of the installation, and both image and sound undergo a series of transformations in real time that imbues the grey boxes with a mysterious expressive power that has political undertones. The visitors act, but also tele-act; navigating in real space and in virtual space, they are guided by their senses of touch, balance, hearing, proprioception and sight. These usual integration mechanisms of their perception are altered by the afflux of multimodal and heterogeneous sensory information, which is not limited to the experience of the real. Instead, the experience oscillates between real and virtual, while the manipulated objects act as limits and physical constraints, but also as tools and partners of the imaginary. The sensorial displacement gener-

Produced by: kondition pluriel in coproduction with the Digital Synesthesia Group With the support of: Conseil des arts et des lettres du Québec (CALQ) Conseil des arts du Canada (CAC) Concept and idea: Martin Kusch and Marie-Claude Poulin Artistic direction: Martin Kusch and Marie-Claude Poulin Interactive design and media content: Martin Kusch and Johannes Hucek Visual programming and technical setup: Johannes Hucek Sound: Alexandre St-Onge

ated by the experience of the installation, a doubling of personality, does not allow the visitors’ body image (the corporeal schema) to stay intact: rather, it becomes dislocated to reincarnate itself in another form, in a synesthetic autopoiesis. The “construction blocks” of At Play, symbolic objects of both childhood and the adult world, evoke our enactment of social roles through our functions, or simply by the way we live, in a globalized world waiting to be reinvented. By proposing an inversion of perspective – an attempt to see the world from the point of view of objects, At Play brings into question our existential dilemma and our relationship with things and places. The boxes ultimately become information units, the visitors potential catalysts of change, and the installation a transformation platform.

113

Diver — a cyber-kinesthetic installation kondition pluriel (Martin Kusch / Marie-Claude Poulin)

Diver — a cyber-kinesthetic installation

kondition pluriel (Martin Kusch / Marie-Claude Poulin)

Artistic media: Various options (360˚ immersive full-dome projection installation; dynamic multi-screen installation; giant urban screen application) Primary sensory modalities: Kinesthetics, Vision, Audio 2016

A

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

114

“By crisscrossing assignations between media-da fabric of relationships in which all the elements

It is difficult to understand how the sense of corporeal movement, or kinesthesia, is commonly excluded from the list of the traditional five senses, even though it is essential to our survival. A plausible reason for its omission is its reliance on multiple receptors located deep within the body (muscles, joints, inner ear). Also, unlike the case for the other senses, corporeal movement is not generally perceived consciously. (cf. A. Berthoz) Diver is a transdisciplinary interactive media installation that gives eyes to multiple viewpoints captured from the body of a competition springboard diver. The images, sound and biometric data – like velocity, kinesthetic inertia and gravity – as well as camera-tracking data originating from the diver’s body, are the source-material for the installation. The piece is adaptable to three installation formats: a 360˚ immersive fulldome projection installation; a dynamic multi-screen installation in a gallery setting; and a giant urban screen application. By moving about or by using their smart-phones, visitors and passers-by modify simultaneously transmitted video and audio recordings from the various viewpoints, transforming and recombining the media material, drastically slowing it down or speeding it up. The orchestration of these mobile and dynamic viewpoints offers a myriad of possibilities for composition; in collaboration A kondition pluriel: Diver (Sketch 1), 2016 © kondition pluriel

with other viewers, the participants become the co-authors of an augmented visuo-audio-kinesthetic work. Regarding her video-installation machine-vision, Steina Vasulka said: “When a human being operates the camera, the assumption is that the camera is an extension of the eye. You move the camera the way you move the head and the body” (1974). In her work she mounted motorized mirrored spheres in front of camera lenses, simulating all possible camera movements, and spoke of “freeing the human eye from being the central

115

point of the universe.” Today we have reached a cultural state where technological developments lead to a hyper-media reality and our perception expands beyond the micro- and macroscopic level. The perfect simulation takes over the imperfect reality and we live in a multisensorial state of existence. With eight cameras installed on the surface of a diver’s athletic body, his limbs, joints and skin transform into a myriad of eyes. This policed

kondition pluriel (Martin Kusch / Marie-Claude Poulin)

-data and control-data, we weave an intricate nts in place resonate and answer in organic ways.”

body, a metaphor for the struggle for control of the physical body, becomes “the center of the universe.” The movements of this perfectly controlled and hyper-conditioned body become the motor for delivering an insight into that which we are not able to see or perceive. Diver

kinesthesia into an uncommon union, in which these senses are the extension, or the mirror of each other. This project deals with systems of control and the human body as a metaphor of that control, and it questions our connection to authorship and the autonomy of humans and machines. Data captured, analyzed and choreographed by machines are part of this process. The project reflects on the tension between representation and performance, between the organic and the mechanical and between centralized and distributed control. In Diver, the viewer is invited to explore the materiality of media in real time, and the installation offers a participative multi-sensorial experience of a hybrid reality that incorporates the performative and event-like

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

116

Diver — a cyber-kinesthetic installation

brings into being a hyper-body with multiple body-eyes that become the extensions of the viewer’s eyes. Sight and hearing are coupled with

B

gestures of an athletic body. A blend of human and technical presence, this body has no “real” existence or single identity, no personal history and possibly no real future in our society of surveillance and spectacle. B kondition pluriel: Diver (Sketch 2), 2016 © kondition pluriel

However, the installation apparatus dismantles this anonymous, normative corporeal condition: instead of simply stimulating the observer’s vision in isolation, the extreme slowness of the source images and the rhythm and sequenced timing of the media elements create new narratives, potentially touching the senses and deep-seated memories of the viewer. Paradoxically and unexpectedly, the installation conveys a decolonized kinesthesia close to Walter Benjamin’s flâneur: a trembling, floating, vacillating body; an environment-architecture oscillating between background

Produced by: kondition pluriel in coproduction with the Digital Synesthesia Group With the support of: Conseil des Arts et des Lettres du Québec (CALQ) Conseil des Arts du Canada (CAC) Concept and idea: Martin Kusch Artistic direction: Martin Kusch and Marie-Claude Poulin Interactive design and media content: Martin Kusch and Johannes Hucek Visual programming and media setup: Johannes Hucek and Martin Kusch Sound: Alexandre St-Onge and Martin Kusch Diver: N.N.

and foreground; the experience of the dizziness of falling; images and sounds suspended in time and space. Diver offers an exploration of sensory relations within an elastic notion of time, of perception from the inside rather than the limited perception of the physical exterior of things. In Diver, positioning and tracking technology separates the components of the action, then recombines them in various layers to create relationships between the media-created space, the recorded space, and the real space. The generic athletic body is made accessible to the nonathletic bodies of the viewers via their physical involvement. The repeated images of the dive transform into an inextricable post-human machine performance. What we see is a hyper-networked-body, encased in environments both immediate and distant; a social and autopoietic entity revealing cyber-kinesthesia, the fruit of an integrative digital visuo-audiokinesthetic synesthesia that allows us to experience other possibles.

117

E.E.G. KISS Karen Lancel / Hermen Maat

Karen Lancel / Hermen Maat

Artistic media: Performance, installation Primary sensory modalities: Taste, Touch, Vision, Audio 2016

E.E.G. KISS

A

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

118

“Can I kiss y

Can I kiss you online? Can we measure a kiss and what kissers feel together? Can we transfer a kiss and its intimacy online? Do we want to save our private kisses in a transparent database – to be used by others? In E.E.G. KISS we explore how a kiss can be translated into data. In a poetic, electric environment for kissing and measuring, for synchronizing and merging, we research a new sensory synthesis for networked kissing. A digital synesthetic kissing ritual In E.E.G. KISS we deconstruct and turn around control technologies and sensory perception, to facilitate a new intimate kissing experience. Worldwide, in performances and live kissing experiments with E.E.G. sensors, we invite visitors to kiss. While kissing, their brainwaves are measured and visible in real time as streaming E.E.G. data. These data are projected around the kissers as an immersive environment, a “Kissing Cloud”, creating an aesthetic and spatially immersive experience. All witnesses of the A While kissing, the kisser’s brainwaves are measured and visible in real time as streaming E.E.G. data. Photo: Lancel /Maat 2014; Eye Film Institute Amsterdam  /  Discovery Festival 2014

kiss are invited to be both voyeurs of the kiss and scanners of E.E.G. data. The E.E.G. data are translated to a music score, based on an algorithm, which is further translated to a soundscape. The soundscape is generated by the kissers’ streaming E.E.G. data. In this kissing ritual both kissers and witnesses share a trans-sensory experience of “kissing”, and “E.E.G. data”. Together they feel, see, hear and mirror a communal, networked kiss. The soundscape has been developed with Tijs Ham at STEIM Amsterdam. Intimacy and Big Data: critical reflections A kiss is an obvious case of emotional interaction and synchronizing, spatial nearness, touch, sight, fluids, smell. When our kisses are being quantified, on what data-visualization design do we validate this interaction? Who is responsible for and who will benefit from our quantified kisses? And how can we trust our own kiss data? In E.E.G. KISS we invite

119

Karen Lancel / Hermen Maat

s you online?”

E.E.G. KISS

B

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

120

C

participants to give meaning to the abstract, even mystifying E.E.G. data-visualization, based on the embodied memory of the kiss as a form of “intimate co-creation”. and ‘co-attribution’. Each unique E.E.G. KISS data stream is saved in a database and can be printed showing “a portrait of a kiss”. Share your private kiss Each E.E.G. KISS soundscape is saved in a public database on the artists’ website to be downloaded and shared. Art, Science & Technology E.E.G. KISS emerged from further artistic research that we conducted at Delft University of Technology in the context of Lancel’s PhD trajectory (Promoters: Prof. Dr. Frances Brazier, Dr. Caroline Nevejan) and is further developed with innovative sensor technologies, in cooperation with neuroscientists and partners in art, science and technology. To inspire B Frascati theaters Amsterdam “End of Privacy festival 2016” Photo: Anna Van Kooij

and rethink sustainable socio-technical ecological systems, we deconstruct automated control technologies (such as surveillance and social media, biometric technologies and brain-computer interfaces) to inspire sensitive relations based on intimacy, digital synesthesia, tacit knowledge,

C EEG KISS participatory, shared ritual for a trans-sensory experience of “kissing”, “heartbeats” and “EEG /E  CG data”. Photo: Roy Te Lintelo 2015; (At Gogbot Festival 2015 “Internet of Things  /  Theories of Future Visions”)

bio-synchronization and aesthetic perception. In this way, we explore digital synesthetic “trust systems” based on holistic experience. Our aim is to create Meeting Places and Meeting Rituals for sensitive networked public space, with shared response-ability for the power of reciprocity and synchronizing, through touching, breathing, kissing, dancing, sharing space and presence.

121

Karen Lancel / Hermen Maat D

Notes on the E.E.G. KISS Project by Lancel/Maat. “When I kiss you, I can taste your soul.” (Terri Guillemets) One starting point for discussing Lancel /Maat’s E.E.G. KISS project could much the cultural and regional differences (as brilliantly documented in Lev Manovich’s Selfie City installation) but the technological homogeneity of these selfie moments. The pressures on the subject-as-user to share the compulsive exhibition of the present with others results in

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

122

E.E.G. KISS

be its contrast to the selfie. What is striking about the selfie cult is not so

a perverse update of Vilem Flusser’s apparatus concept in photography theory. What is expressed in the compulsive selfie is not just the template culture of the social media platforms and technical specifications of smartphone lenses and the (default) settings of their software, but also the victory of logistics, with its massive online services, broadband and D Photo: Lancel  / Maat 2015; At Brakke Grond  /  Frascati Theatre Amsterdam (Yami-ICHI and Privacy Festival)

data-centers that all “eat into” the image in an unconscious manner. Instead of the ego presence of the selfie, E.E.G. KISS presents us with a shift towards an inner experience with the other. Lancel and Maat’s techno-performance of colliding wetware captures the intimate act of kissing in a time-based fashion. Their autograph of love emphasizes the social aspect: we never kiss alone. This is no auto-erotic kiss but a collective experiment that goes way beyond the self-centered exhibitionism that defines our social media age. The installation opens up a space of free interpretation. Despite the neuroscientific look and feel, users have to be made aware that the produced graphs of their kiss do not contain an eternal truth. The displayed data are imagined intensities; never medical evidence of whatever. The data visualizations merely represent our obsessive techno-culture. In an epic quest to invent new interfaces and tools that will help us to develop the visual language that defines our era, some artists are willing to take risks — not just with their reputation. They ask the question: what’s a quantified kiss? The risk in the case of E.E.G. KISS is an obvious one and goes beyond the danger of being appropriated by “the system”. What if

In collaboration with and the generous support of: Digital Synesthesia Group, University of Applied Arts Vienna; Mondriaan Foundation (Vrije beurs praktijverdieping); STEIM Amsterdam; Baltan Laboratories Eindhoven; Fourtress Eindhoven; Holst Centre Eindhoven (sponsor E.E.G. headsets IMEC); University for Technology, Delft / Participatory Systems Initiative; Discovery Festival, EYE Film Institute Amsterdam; Neuro-engineering lab Tsinghua University Beijing; Social and Affective Touch Research Group TNO (Netherlands Organization for Applied Scientific Research); Artists-in-residency TASML Tsinghua Art and Science Media Laboratory, Tsinghua University Beijing

our kisses are stored in databases against our will and analyzed by algorithms owned by banks that decide on home loan applications? An all-too-passionate kiss can indicate that the wild romance will end up in a dramatic divorce. A flat and boring kiss will transmit the same message. This puts the motive of the “stolen kisses” in another context. All too easily, networked data facilitate the most absurd readings into a situation. Rebel souls will arise and claim that united spirits will never be documented. Why store an act of human intimacy in a machine, when the real issue is our acute loss of memory — of any experience? How can we train our bodies to, again, store our vital data? Instead of administrating the last remaining expressions of us, being human, artists arise against the technological elimination of our instincts. The quantified artist is the problematic figure here, the willing researchers that smoothen out the clean agenda of the virtual sex industries. Instead, let’s investigate together how the fluids, flowing together, can short-circuit our cynical architectures of power. As Billy Holliday said: “A kiss that is never tasted, is forever and ever wasted.” (Geert Lovink)

123

The Flying Umbrella Project Alan Kwan Artistic media: Aerial Robotic Performance Primary sensory modalities: Kinesthetics, Touch, Audio 2016

The Flying Umbrella Project

Alan Kwan

“An erotic synesthetic performa

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

124

ance that takes place in mid-air.”

The Flying Umbrella Project is an outdoor surreal erotic performance that takes place in mid-air. The performance consists of two robotic flying creatures in the form of umbrellas mating in the air. Created with customized flying ma Artist: Alan Kwan Artistic & Engineering Advisor: Bjorn Sparrman Engineering Advisor: Roger Wang Sound Designer: Jose Alejandro Rivera In co-operation with TONSPUR Kunstverein Wien (Artistic director: Georg Weckwerth) / Q   21 (at MuseumsQuartier)

chines, control algorithms, and animatronics, these umbrella robots kiss, embrace, caress, and copulate with each other in the air. At the same time, specific sound frequencies are generated digitally from these body motions and intimate contacts, allowing audiences to not only see but also hear the sensuality and lust of these artificial life organisms. The performance therefore explores the idea of robot pornography, and also investigates the use of flying robots in creating motion-sound crossmodal perceptual experiences for artistic aerial performance.

125

Bestiary for the Minds of the 21st Century: Genomic Opera A data-mining, synesthetic, 3D-printed genetic opera Marcello Mercado

Bestiary for the Minds of the 21 st Century: Genomic Opera

Marcello Mercado

Artistic media: 2-channel video installation, artist’s book, public audio interaction Primary sensory modalities: Kinesthetics, Touch, Vision, Audio 2015

A

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

126

“Digital Synesthesia is a data mining process, trans

We live in a biodiversity crisis: A massive daily loss of biological data has led me to think about a genomic opera that could preserve/archive fragments of thousands of prokaryotic and eukaryotic genomes (like a Noah’s Ark) that can be combined and transformed into musical notations in the form of tactile 3D-printed objects. The aim of the art project is to create 3D-printed objects based on various mixed genomes, as well as to explore the possibilities of 3Dprinting errors (glitches). Fragments of the genetic information of different types (mammals, insects, fungi, bacteria, etc.) are randomly mixed, and by means of algorithms transformed into audio files which are then converted into 3D-printed objects. These environment-friendly 3Dprinting filaments, combined with glitches and incomplete 3D-printed remains, make up the score of the musical notations. I researched the available genome databases like Ensambl, Genome, Wormbase, Flybase, etc. to mix and cut eukaryotic and prokaryotic genomes in random combinations. The nitrogenous bases (as the letters A, T, G, C) are read by a software that transforms text into audio files. These are transformed into 3D objects, which are again printed in plastic with a 3D printer and pasted onto paper as music notations. The generation of AUDIO and VIDEO through TACTILE actions involves audience participation: The audience interacts by rubbing the 3D-printed objects with a piezoelectric microphone connected to a transductor to visualize new sounds. In parallel, each page of the opera is photographed and scanned with sonogram software in order to be converted into sound A ABS-plastic (ABS) and polylactic acid (PLA) (bioplastic) on polymethyl methacrylate (PMMA), copper colored, 2015 Page from the Opera-Artist’s Book: 3D printing errors (glitches), 3D remains and data mining process on genomes from mammals, insects, fungi and bacteria converted into plastic 3D-printed objects (40   cm x 60   cm x 30   cm)

archives.Thus, the final performance of the opera is a combination of: • 3D-plastic-printed genome-data-mining-instruments, performed live by a musician rubbing the 3D printed objects with piezoelectric microphones, • Recorded mixed genome fragments, • Sonogram audio files, and • The score that instructs the performers during the entire duration of the piece throughout the four acts.

127

Bestiary for the Minds of the 21 st Century: Genomic Opera

Marcello Mercado

anslated into the space by means of algorithms.”

B

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

128

On Marcello Mercado’s Genomic® Opera: Bestiary for the Minds of the 21st Century For a long time dualism served as a convenient substitute for thought. We distinguished between artists whose creative process is driven by their minds and those who rely on their gut instinct. Defined in terms of media theory: A software and a hardware approach were interpreted as oppositional and frequently antagonistic positions. This was the 20th century analogy to the body-soul dualism of the European Middle Ages which branched out into different guises in the modern age. The subversive perspective of alchemy, for example, vehemently contested this dualism even in the pre-modern era. The tradition of pre-Socratic philosophy in Greek antiquity supposed that the soul was flesh through and through, where matter was considered nothing more than modified spirit embodied in various forms and conversely spirit was only a subtle formulation of matter. The disciples and sorcerer’s apprentices of alchemy developed this thought further into an experimental practice and an enigmatic Hermetic paradigm. The lapis philosophorum, the philosopher’s stone, which is thrown onto stubborn matter in the last B ABS-plastic (ABS) and polylactic acid (PLA) (bioplastic), gouache and pencil on transparent paper, 2015 Pages from the Opera-Artist’s Book: 3D printing errors (glitches), 3D remains and data mining process on genomes from mammals, insects, fungi and bacteria converted into plastic 3D-printed objects (40   cm x 60   cm x 30   cm)

stage of transmutation to refine and instill it with metaphysical meaning, is both: the powdered matter of transformation and an algorithm in the sense of an order which was distilled on the basis of scientific experiments. Argentinian artist Marcello Mercado is working in such a subversive tradition, exhibiting radical consistency as much as exuberant energy and ingenuity. In a first performance which I saw almost 20 years ago, he confronted the complex electronic and digital machines of a post-processing unit costing millions with his naked body and the urges of the mind imprisoned within it and clamoring to get out. With all the power of his sensory perception and experience he dedicated himself to generating worlds of images and sound and developed with Das Kapital [Capital] a gigantic opus which cannot end, because it has become an obsession and is thus not so

129

D

E

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

Bestiary for the Minds of the 21 st Century: Genomic Opera

Marcello Mercado

C

130

much organized in a linear as a dynamic and chaotic fashion. The visual and acoustic language that M2 invented for his extremely idiosyncratic interpretation of the Kapital as a single political economy of the body is a continuous process of transforming data and relations into sensations and the aggression to be experienced; all of this has proliferated excessively, already yielding more than 40 hours of recordings. Simultaneously to the transformative work of Das Kapital, M2 has over the last ten years experimented with genomic material. His studio has be

come a biological and chemical research lab; he dug outdoors, extracted the lifeblood of plants and animals and started to obsessively break them

C Artist’s Book

down into their individual components and combine them, much like he does with data clusters in his computer and the recordings of his analog sound equipment. Gay film legend Fassbinder and heterosexual Fluxus

D Pencil on transparent paper, 2015 Page from the Opera-Artist’s Book: 3D printing errors (glitches), 3D remains and data mining process on genomes from mammals, insects, fungi and bacteria converted into plastic 3D-printed objects (40   cm x 60   cm x 30   cm)

legend Vostell never came together in reality. M2 mixed their genetic information using extracts distilled from flowers growing on their graves and merged them in drawings which he produced using the extracted genomic material, blurring the boundaries between virtuality and reality. M2’s Bestiary for the Minds of the 21st Century is the culmination of combining various strands of his work over recent decades. Much like in the case of Kapital, which he composed as an oratorio and for which he

E Gouache, acrylic paint and pencil on transparent paper, 2015 Page from the Opera-Artist’s Book: 3D printing errors (glitches), 3D remains and data mining process on genomes from mammals, insects, fungi and bacteria converted into plastic 3D-printed objects (40   cm x 60   cm x 30   cm)

wrote several libretti, he advances his opera experiment as an aesthetic genre in which precision and passion interact in an extraordinary manner. Sound is translated into visual patterns. In contrast to the legendary figures of Ernst Florens Friedrich Chladnis from the 18th century, he renders the temporal variability and audible intervals not merely on canvas, but in 3D-objects. Once transduced into data files, the sounds are printed out as oblong sculptures in bright colors, which on a notation can be experienced as wavelike information clusters in the truest sense of the word. Using piezoelectric microphones as an interface, the opera audience is able to scan the surface of the material. The experimental apparatus in turn converts the sound thus produced into graphical structures, producing an entirely different experience of the original composition for the viewer, as the result of the tactile interaction now visualized in the resulting audio/visual representation. Another interpretative aspect is that with this Genomic Opera M2 seeks to contribute to evolving the diversity of species in our biological world, which he sees as threatened by the contemporary drive to standardization. However, the knowing of this is not a necessary prerequisite for the aesthetic and intellectual enjoyment of the composition. Much like in Das Kapital, his impetus toward an undisciplined heterogeneity becomes a tightrope act, unfolding above the abyss of our reality, oscillating between program and imagination as well as between rule and ingenuity rebelling against censorship. (Siegfried Zielinski)

131

Sound Calligraphy Ulla Rauter

Sound Calligraphy

Ulla Rauter

Artistic media: Performance, drawing, live-sonification Primary sensory modalities: Vision, Audio, Kinesthetics 2015 – 2016

A

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

132

“The translation from one medium into anoth process, which is able to reveal new aspects

The project consists of calligraphic handwriting containing the spectral sound information of spoken words, which can be made audible by techniques of sonification. During a live performance, I draw calligraphic handwriting which is trans-formed into sound by means of a special scanner (camera and computer-software). I work at the interface between sound and fine art, exploring processes of transformation. In earlier works I focused on a special kind of sound visualization, on spectrographs (also called voiceprints): A spectrograph is a 2-dimensional time-frequency analysis of a sound. I am not only fascinated by the aesthetics of these images but also by their function as a medium – like a code, they contain a certain message as well as the information about the original sound. With certain technology it is possible to convert them into sound again and recover their acoustic contents.

In my work Sound Drawings (2010), I began to create hand-drawn spectrograms: I reversed the process of sound-visualization and painted spec-

A Sound Calligraphy, Performance (Sketch)

trographs of a human voice on the wall, which afterwards were made audible through an inverse Fast Fourier transformation.

133

Ulla Rauter

other is like an alchemistic ts of the source material.”

Sound Calligraphy

B

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

134

C

My interest lies in the fact that the combination of manual drawing and digital technology is able to create an artificial voice and the perception of speech. Apart from the poetic performative aspect of drawing voices, human speech and its perception are part of the focus of my interest.

My drawings are in fact only layers of frequencies that imitate a human voice. It is not much more than noise and a cluster of sine waves, and it

B Sound Drawings, 2010

C Sound Calligraphies (Sketches)

is artificial. But at a certain point in time, we can start to recognize words, maybe a sentence; we start to hear human speech. Due to the constructive behavior of the brain, we search for meaningful messages within the noise that we hear.

135

facades Ruth Schnell

Ruth Schnell

Artistic media: Audiovisual animation, immersive Fulldome projection Primary sensory modalities: Vision, Audio, Kinesthetics (Balance, Visceroception) 2016

facades

A

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

136

“Different motion parameters interact. The result is a dramatur

A facades (2015), performed in the context of Liminal Spaces – E /M /D/ L, Satosphère, Montreal Photo: Johnny Ranger

Constructed from arrangements of point clouds, a virtual urban scene is developed in facades. The viewers seem like flâneurs, following streets that are flanked by classical facades. The movement of the viewers is virtual and is generated by the movement of the projected image space. facades is an audiovisual animation, conceived for performance in a Fulldome. Fulldomes are immersive, dome-based projection environments which – as an interface – allow the full and multimodal integration of the viewers in a multi-sensory, potentially interactive experiential space. Ruth Schnell makes use of the properties of this medium, which is comparatively young in the domain of the arts, to explore and challenge traditional concepts of space, reality and perception – a central theme of her art. Ruth Schnell has designed a dramaturgy of disintegration and reconstruction. The seemingly “organic” facades and streetscapes appear elastic; diaphanous rows of houses overlay each other, panoramas may unfold. The viewer can move virtually in a horizontal fashion or into the depth of the configuration. The individual motion of the projected content

Concept and idea: Ruth Schnell Artistic direction: Ruth Schnell Media content and animation: Nikola Tasic Media setup: Martin Kusch, Johannes Hucek Sound: Alexandre St-Onge, Marie-Claude Poulin

simulates both spatial dynamics as well as a first-person perspective. Sound accompanies this moving image. It is based on the recording of the text No Such Agency, published in 1986 by media theorist Friedrich Kittler about the NSA. Whispered fragments of the text are acoustically placed in the projection, seeming to drift past, and coming together in the virtual space.

137

Ruth Schnell

rgy of disintegration, an experience of instability.”

facades

B

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

138

C

The image content remains open toward the top and bottom; without the reference points of “sky” and “ground”, spatial delineation is undermined. The presented immersive experience is also an experience of instability. The imagery of facades has its starting point in 3D models of city buildings, developed using a photogrammetric method. Here, through several calculations and various software programs, 3D information B facades (2015), performed in the context of Liminal Spaces – E /M /D/ L, Satosphère, Montreal Photo: Johnny Ranger

on an object is derived from only two-dimensional images. The 3D models, generated from point clouds and combined with a 3D graphics program to form streetscapes, allow changes in perspective and various simulated movements. facades was originally developed for dome projection in Montreal’s Satosphère, a Fulldome 18 meters in diameter with a 360-degree

C facades (2015), animation still Photo: Nikola Tasic

expansion of the image on the horizon and 210 degrees at the zenith. (Patricia Köstring)

139

MotU #4 – #6 Ruth Schnell

Ruth Schnell

Artistic media: Light objects Primary sensory modalities: Vision, Kinesthetics 2011/2016

MotU #4– #6

A

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

140

“The habitual process of visio

B

The term “to see” is often used in common speech synonymously with “to understand” or “to grasp”. The phrase “If I am seeing this correctly” is used to signify understanding in consensus with others, while those who “don’t believe their eyes” experience something unexpected and untoward. But in general we do believe our eyes; so much so that we consider the information our visual senses present us with to be an accurate image of the A MotU #3 (2011), solo exhibition, Galerie Grita Insam, Vienna Photo: Birgit and Peter Kainz

present moment, to be reality.

B MotU #2 (2011), solo exhibition, Galerie Grita Insam, Vienna Photo: Birgit and Peter Kainz

emerge from three flickering LED beams and float hologram-like in front

The work MotU #4 – #6 from the series Mirrors of the Unseen challenges this supposed credibility of the visible. The habitual process of seeing is infiltrated by words and icons made of light; they seem to of the surrounding architecture. Nothing can get in the way of these translucent (written) images. They are present, and yet at the same time do not exist: only individual perception makes them accessible to the viewer. Ruth Schnell started developing so-called “light sticks” at the beginning of the 1990s. Since then, the works created using these media have been technically advanced; they befuddle our idea of perception.

141

Ruth Schnell

sion is infiltrated by words and icons made of light.”

MotU #4– #6

C

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

142

The light stick installations make use of the inertia of the human eye: The photoreceptors of the retina cannot resolve separate frames at a rate of more than about 20 frames per second. This is the reason why we see film as film, why the annular motion of a flashlight in the dark appears to describe a closed circle of light. For the display on the light sticks, words, icons and also schematic photographic images are broken down into points that are successively shown on the vertical columns of the beam. The characters thus converted into a temporal sequence of highfrequency pulses then become readable so long as the viewer forgoes habituated reception patterns practiced through the act of reading. What this necessitates is a free-floating gaze, an observation in passing. While moving, the viewer takes in an afterimage, thus seeing several past situations as a continuous present event. The performative act of moving alone creates this “present” and thus also the work as a collaborative process. In MotU, the viewer is therefore particularly involved in the creation of the work. The icons and images on the sticks alternate at a certain frequency. The afterimage is distorted in direct relation to the individual’s perspective. There is no collective viewing of the work; what is seen is an individual experience rather than something tangible or verifiable. Site- and situation-specific works and series of works with light sticks C All Targets Defined (2006), solo exhibition, Galerie Grita Insam, Vienna Photo: Birgit and Peter Kainz

are created for very diverse contexts, particularly responding also to the viewers’ expected movement patterns. Large-scale light stick installations have been conceived for public space with works such as Floating Signs (Bregenz harbor) or the interactive installation laut und leise (Dornbirn indoor pool). Several small beams, arranged in relation to students’ and employees’ rhythms of movement, are being used in the commissioned art projects for HTL Bregenz (polytechnic) or the EnBW’s headquarters in Stuttgart.

Concept and idea: Ruth Schnell Artistic direction: Ruth Schnell Media content: Ruth Schnell, Lea Schnell, Patricia Köstring Programming and media setup: Stefan Istvanits, Alexander Pausch

With regards to content, the works weave a fabric of thematic representation. The words and icons projected onto the sticks for MotU #4 – #6 include terms from the field of synesthesia which are associatively condensed through sequencing and rhythmization. (Patricia Köstring / Ruth Schnell)

143

Topography of Movement Ruth Schnell

Topography of Movement

Ruth Schnell

Artistic media: Dynamic projection Primary sensory modalities: Vision, Audio, Kinesthetics 2016

A

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

144

“The dynamic projection does not interpret the architectural space as background

A Gegen die Zeit (2008), Tiefenrausch exhibition, a project by Offenes Kulturhaus and Linz09, Linz Photo: Otto Saxinger

A hand moves, it explores its surroundings, traces the surface on which it lies, it acts. Based on this actually simple cinematographic process, in Topography of Movement conventional perspectives of the two-dimensional (the motion picture) and the three-dimensional (the real environment) are altered. The relationship between image projection and screen changes, the space is reconstructed by the cinematic sequence. For her dynamic image projections, Ruth Schnell works mostly with computerized projection mirrors. The image content itself is cropped and projected without a background. It moves about in manifold ways: In Topography of Movement projected hands perform an activity. The

Concept and idea: Ruth Schnell Assistance: Patricia Köstring Artistic direction: Ruth Schnell Media Content: Ruth Schnell, Patricia Köstring Postproduction: Alexandre St-Onge Camera: Azalea Ortega, Nikita Zhukovskiy Performer: Marie-Claude Poulin, Nikita Zhukovskiy

sequence in turn is dynamized, accelerated or counteracted via the movements of the projection mirror, depending on how the two movement patterns come together. Digital image production and a spatial reference system intertwine via these overlapping movements. The image projection does not interpret the architectural space as a background or screen, but as an element of a new image formation; the projection itself is conceived not as a frame, but as an independent movement of unfettered images that can theoretically occupy any part of the room. So the real space with the help of the moving projection is newly modeled because, depending on the projection angle and surface structure, distortions and fragmentations of the image content arise in the contrast between projection and architecture. The multiple projection (a right and a left hand), the scaling of the projected scene, as well as the implied tilt of the camera angle from top view to a side view override conventions of spatial order. If the left or

145

Topography of Movement

Ruth Schnell

und or screen, but as an element of a new image formation.”

B

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

146

C

right hand point to an absent body, a subjective viewer perspective is made impossible by the high magnification of the image content. The scaling changes the spatial reference system. The viewer hears the original sounds that arose during the recording simultaneously to the motion of the image: wiping, knocking, patting sounds. The sound thus contributes to the spatialization of the image content. With this environment, Ruth Schnell is continuing her artistic research in the field of dynamic projection. Hands and their movements were already addressed in two previous works: Gegen die Zeit, originally conceived in 2001 for the secularized Johanneskirche in Feldkirch and B Territorism (2002), facade projection commissioned by Kunsthaus Bregenz, Bregenz Photo: Ruth Schnell C Gegen die Zeit (2001), installation, Johanniterkirche, Feldkirch Photo: Ruth Schnell

shown again at the 2008 Tiefenrausch exhibition in Linz, projected a woman’s arm with a scrubbing brush and presented a textual ambiguity between “covering one’s tracks” and “critique of reproductive work.” The multi-piece video projection Territorism, developed for the Kunsthaus Bregenz (KUB), displayed projections on the glass entrance and side facade of the KUB and the facade of the neighboring Landestheater each night in the summer of 2002. Two hands were projected; one held a toy tank, the second seemed to rest on an illusory floor. This gave rise to a simulated spacial structure that intertwined the solitary building of the KUB with its architectural environment. (Patricia Köstring / Ruth Schnell)

147

IN_SIDE VIEW Jeffrey Shaw / Sarah Kenderdine

“ In_side a pataphysical allegory for the multi-sensory

Jeffrey Shaw / Sarah Kenderdine

Artistic media: Immersive installation Primary sensory modalities: Orality (Taste), Touch, Vision, Kinesthetics 2015

IN_SIDE VIEW

A

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

148

encounter between the body and the perceptual imaginary, between nature, manufacture and ataxia.”

A Panoramic photos of Angkor Wat: Jeffrey Shaw and Sarah Kenderdine Panoramic 3D photos of a fire-damaged hardware store in Sydney: Jeffrey Shaw and Volker Kuchelmeister

“The huge snake roots, the vaster / serpent arms fallen / octopus over the roof / in a square courtyard – curved / roofcombs looked Dragon-back- / stone-scaled / As frail as stone is, this harder wooden / life crushing them // with the cricket-glare and parrot / squads walking across the roof / – last nite full moon in misted heaven / and slow girl dance bent elbow and insp[i]ring / fingers snaking it thru the middle –“ (Allen Ginsberg: “Angkor Wat”, 1963) 1 https://en.wikipedia.org/wiki/ Circle_of_confusion

“In forging the narrative of things, senses are separated to make sense of the myriad impressions” (Virginia Woolf). In IN_SIDE VIEW senses are senselessly conjoined, the world put back together, “an incessant shower of innumerable atoms” (ibid), seeing oneself as another, in side out, in the circle of confusion1, amongst trees / stone-scaled / snake roots // eyed / tongue in cheek / switchback // past / future! / present // cable / tie // wrap / around // heady / giddy / techy / tacky / vroom // memory / aromatic / terror // ruination / fervid bedlam. The where-with-all in_side a

Application software: Leith Chan Hardware: Samsung Gear VR, Conceptus tongue switch

pataphysical allegory for the multi-sensory encounter between the body and the perceptual imaginary, between nature, manufacture and ataxia.

149

Transmission+ Interference David Strang / Vincent Van Uffelen

Transmission + Interference

David Strang / Vincent Van Uffelen

Artistic media: Sound-light installation Primary sensory modalities: Vision, Audio, Kinesthetics 2016

A

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

150

“What if one could utilize someone’s unique sensing

This installation was created by David Strang and Vincent Van Uffelen and is part of their larger collaborative project Transmission+Interference. The artists explore the creative potential within light as both the creator and transmitter of sound and actively move away from creating visualizations of sound towards an audiovisual experience based on resonance and feedback loops between light, sound and object. Using performances, workshops, and installations they attempt to learn about information, disturbance, noise, glitch, communication, as well as the ephemeral relationships that emerge from both their and the audience’s engagement. The artists aim to reveal the complexity of these fleeting experiences that occur in the moment, those glitches that happen somewhere between physical sensors and mental representation that can’t be recorded, repeated, or simulated. Transmitter > Multi-Modal Crossover > Receiver Consisting of three parts, the interactive installation investigates the potential for using digital devices to create a synesthetic communication system. The installation draws on the different experiences of synesthetes and uses this to explore a messy and playful method of communication. Using digital protocols of ASCII and frequency conversion, letters are A LED and solar cell sound transmitter and receiver, Workshop in Plymouth 2013

encoded into a combination of a light color and a sound frequency (e.g., “A” is transmitted as a yellow light and a 50 Hz sine wave) allowing a simulated synesthete to decode the text that has been transmitted through space. However, to get closer to a synesthetic communication process, the classic Transmitter –> Receiver pattern must be interrupted by a multi-modal crossover that mimics a synesthete’s unique experience of light and sound. The visitor is able to adjust the multi-modal crossover to gain further understanding of a synesthete’s unique experience whilst interfering with a potentially flawless communication flow. By expanding the communication system across three distinct sections, the installation develops defined territories or zones through and

151

g ability? Would this bE AN EfPectiVe d00l T0I sTckr3ly c$m_uEicR7s!”

Light amplitude modulator

Arduino

Potentiometers

Mini Computer

Amplifier & Speaker

Micro Computer

Microphone

LEDs Colour Sensor Screen

TEXT TO SEND

Screen

Solar Cell

RECEIVER

Transmission + Interference

B

David Strang / Vincent Van Uffelen

Micro Computer

CROSS-MODAL MODULE

TEXT RECEIVED

TRANSMITTER

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

152

C

over which information (noise and signal) flows. Although these sections of the system appear as physical material in the space, the artists are actively exploring the spaces between the physical parts. It is here that we encounter the resonance within the system; where neighboring sections meet through the interference of external noise (light and sound) and through which new meaning potentially begins to emerge. The system encourages a nomadic movement of information across a technological plateau that is initially understood as linear, allowing for the spreading, leaking and disruption of information across undefined routes. What if one could utilize someone’s unique sensing ability? Would this be an effective tool to securely communicate? After all, one could covertly send hidden messages in plain sight that only an intended receiver could decode (steganography). Such a system of transmission could, in the odd event, be learnt and understood by someone able to perceive the blurring of these senses and, with practice, the user could claim our proposal as their own efficient, albeit abstract, working communication tool. B Schematic of the Transmission + Interference installation

Furthermore, the installation is a cybernetic experiment tapping into the unknowable and becoming. The physical installation is also an enquiry into practice in its own right and aims to raise questions about our relationship with technology, our co-evolution, and the blurry lines that

C Laser light sound transmission, Performance Athens 2013

our minds try to define as borders between the self, the body and other technological systems.

153

I am Sound Tamiko Thiel / Christoph Reiserer

Tamiko Thiel / Christoph Reiserer

Artistic media: Interactive installation Primary sensory modalities: Vision, Audio, Kinesthetics 2016

I am Sound

A

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

154

“I see. I hear. I am. I am

On “I am Sound” The installation is an embodied sensory experience, explain Tamiko Thiel and Christoph Reiserer, co-authors of I am Sound. The collaborators first worked together in 2003, creating a real-time sound/video improvisation, secret movements, which involved several musicians, surveillance and infrared cameras, and a live projection behind the musicians. Both artists have continued to develop their respective practices and are recognized for their innovative approach to experimental media. Thiel has produced internationally recognized works in the virtual realm, especially interactive 3D and AR (Augmented Reality) environments. Reiserer is an accomplished composer, performer and creator of sound installations. He has continuously stretched the boundaries of sound and image by using computer software and electronics. Recognizing the relationship between sound and image, and the influence this element has had on the moving image, these artists have gone beyond the typical strategy of creating a soundscape to accompany and enhance an image (or vice versa), in order to join sound and image in a unified statement. They use the image itself as the source of sound, and that image is obtained by data sourced from the viewer’s face. This A I am Sound, test of installation lighting. Tamiko Thiel and Christoph Reiserer, 2016 Mixed media installation (custom metallophone, video, audio, electronics). Dimensions: 4   m wide x 3   m high x 7  m deep (approx.).

informational interaction, the bringing in of the viewer into the work as an essential element, produces a new set of opportunities. The viewer, who is the source of what is seen and heard, is the main component of a finely tuned set of factors. To understand the dynamic of I am Sound, it’s important to appreciate that the installation is in fact a musical instrument system. Presented as a darkened room with a pool of light indicating where the visitor should stand, it is composed of three vital parts: hardware, software and the viewer. The hardware consists of a “curtain” of 12 aluminum plates which are tuned to vibrate at frequencies which produce specific tones and their harmonics. The plates function as both loudspeakers and a projection

155

Tamiko Thiel / Christoph Reiserer

am Image. I am Sound.”

I am Sound

B

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

156

C

screen. A small camera and microphone captures information from the viewer; a willing participant who, facing the camera and microphone, looks forward to where their face is projected in front of them onto the plates. The visible image is how the machine interprets the visual information received: grey, pixelated, in jittery motion, fragmented and framed by green lines as the algorithms select and transform this mirror of the self B I am Sound, shadow test of prototype #1. Tamiko Thiel and Christoph Reiserer, 2016 Mixed media installation (custom metallophone, video, audio, electronics). Dimensions: 4   m wide x 3   m high x 7  m deep (approx.). C I am Sound, metallophone prototype #2 back view: transducers and brackets. Tamiko Thiel and Christoph Reiserer, 2016 Mixed media installation (custom metallophone, video, audio, electronics). Dimensions: 4   m wide x 3   m high x 7  m deep (approx.).

into data. Converted into data, the face, captured as a lowresolution surveillance image, is digital information which vibrates, emitting unique, harmonious audio created specifically from the data obtained from each individual facing the camera. What is revealed, an augmented, amplified selfie portrait, becomes a personal musical “voice” of the subject with interconnected harmonic tones. Underpinning this work is a sense of reflection. Even though it is presented as a challenge, it is not a game but an opportunity for information to be experienced differently. The face, one’s individual identity, is typically small and exposed to the world. But this selfie is large, and seen directly by the participant who can alter and explore new and varied compositions. I am Sound invokes the multiple modes of visual, aural, kinesthetic and spatial senses and generates a contemplative customized musical and visual encounter with one’s self. (Kathy Rae Huffman)

157

Data Music Peter Weibel

Data Music

Peter Weibel

Artistic media: Interactive audio-visual installation Primary sensory modalities: Touch, Audio, Vision 2016

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

158

“Digital music turns every object sources. Each object can simulat

A notation is a fixed instruction written on sheets of manuscript paper. It is a kind of analog programming that tells a human being what to do; e.g., touch the black and white keys of a piano in a prescribed way. Whereas a program tells a machine what to do, a notation tells a human being what to do. These notational commands are written in arbitrary signs for interpretation. In contemporary music the interpreter is given more freedom, more degrees of freedom of interpretation. Today, in the age of digital music, notation has changed its character tremendously. The composer tells the computer in a prescribed way what to do, he writes a script for the computer which becomes an interpreter with a lesser degree of freedom. Digital music deals with data instead of tones and notes. The axiom of digital music is: any word, any sign, any image, any number, any tone can be transformed into data and any data can be transformed into any numbers, sounds, images and texts. Under these circumstances notation no longer serves as only an instruction but becomes an instrument. In a traditional musical set-up, a music stand – which is an empty frame made of iron or wood, a device to hold sheets of manuscript paper on which notes are written or printed – is placed in front of the musician. The musician, or alternatively the interpreter, reads these notes as commands telling him how to perform his instrument. The classical performance therefore consists of four parts: the instrument, the music stand, the performer and the invisible composer present by the notation. In the digital age, however, the notation becomes a physical process, a kind of operative ontology. Based on

159

Data Music

Peter Weibel

ect into a musical instrument. Digital music turns any object into sound late each instrument. The computer becomes a universal composer.”

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

160

digital logic, I constructed an interactive installation. The set-up comprised a music stand shaped like an empty frame with five staves. These five staves consist of five horizontal strings building a kind of abstract geometrical guitar (in a cubist way). The strings are attached to a computer in which several musical compositions are stored, e.g., famous rock riffs. Loudspeakers are furthermore connected to the computer. This setup enables the visitor to pull these five strings, which are the “notation” and the “instrument” at the same time, whereby he becomes the “composer”, “interpreter”, and “performer” of the music. As the music is stored physically as data in the computer, i.e., pre-programmed, we could say that the visitor performs music through an interface (music stand), which is data-programmed by an algorithm. In addition, the performance of the player and of the visitor respectively is recorded by a KINECT camera and projected onto a screen, or alternatively displayed by a monitor. As the images of the performance are purely data, the configuration of the visual data can easily be influenced by the configuration of the acoustic data. Sound and image are synchronized with each other. Thus, whenever the visitor is in front of the music stand, he sees himself projected or displayed in a normal, mirror-like way. The very moment the visitor starts performing with the music stand, his image becomes distorted and appears as a flow of visualized acoustic data. Classical synesthesia is a phenomenon organized by the human brain and is a subjective experience. Digital synesthesia is an objective phenomenon, organized by machines and media.

161

Data Music

Peter Weibel

This book chapter introduces 15 DIGIT been produced by 17 renowned media arts-based research process. The digit mental set-ups for the artistic research Texts, sketches and pictures provide in of the works in the research context o

Software development: Nikolaus Völzow Sound design and interface design: Derek Hauffen

ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA

162

TAL SYNESTHESIA artworks which have a artists in the course of a three-year tal artworks have to be seen as experih of cross- and multi-modal perception. nformation about the artistic concepts of DIGITAL SYNESTHESIA.

163

DIGITAL SYNESTHESIA EXHIBITION Part 3

This book chapter contains the photog SYNESTHESIA exhibition at the Angew which took place from March 11 to Ap which were commissioned by the Digi the international DIGITAL SYNESTHES

164

graphic documentation of the DIGITAL wandte Innovation Laboratory in Vienna, pril 8, 2016. On display were 14 artworks ital Synesthesia Group in the course of SIA Call in 2014.

165

DIGITAL SYNESTHESIA EXHIBITION

166

A Anke Eckardt VERTICAL 2

167

Alan Kwan The Flying Umbrella Project

DIGITAL SYNESTHESIA EXHIBITION

168

Peter Weibel Data Music

169

Ruth Schnell Topography of Movement

DIGITAL SYNESTHESIA EXHIBITION

170

Marcello Mercado Bestiary for the Minds of the 21 st Century: Genomic Opera

171

R uth Schnell MotU #4–#6

DIGITAL SYNESTHESIA EXHIBITION

172

Karl Heinz Jeron Space Time

173

Jeffrey Shaw/Sarah Kenderdine IN_SIDE VIEW

DIGITAL SYNESTHESIA EXHIBITION

174

Karen Lancel / Hermen Maat E.E.G. KISS

175

Anke  Eckardt VERTICAL 2

DIGITAL SYNESTHESIA EXHIBITION

176

Ulla Rauter Sound Calligraphy

177

Tamiko Thiel / Christoph Reiserer I am Sound

DIGITAL SYNESTHESIA EXHIBITION

178

kondition pluriel At Play

179

David Strang / Vincent Van Uffelen Transmission+Interference

DIGITAL SYNESTHESIA EXHIBITION

180

181

This book chapter contains the photog SYNESTHESIA exhibition at the Angew which took place from March 11 to Ap which were commissioned by the Digi the international DIGITAL SYNESTHES

Ruth Schnell facades

DIGITAL SYNESTHESIA EXHIBITION

182

graphic documentation of the DIGITAL wandte Innovation Laboratory in Vienna, pril 8, 2016. On display were 14 artworks ital Synesthesia Group in the course of SIA Call in 2014.

183

184

186 – 197  Authors’ Biographies 198 – 199  Imprint

185

Authors’ Biographies

Gerald Bast Following graduation in Law and Economics at the University of Linz, completed a doctorate in Law (1974). Head of the Federal Ministry of Science and Research Department for Organizational Law and Principle Issues regarding the reform of universities and art colleges (1991– 1999) and consultant for the Ludwig Boltzmann Society for Scientific Research (1992– 1999). Rector of the University of Applied Arts Vienna (since 2000); deputy chairman of the Federation of Austrian Universities; spokesman for the rectors of the Austrian Universities of Art, and vice-president and member of the presidium of the Conference of Austrian Universities. Member of the scientific advisory board of the “Zeitschrift für Hochschulrecht, Hochschulmanagement und Hochschulpolitik” and member of the board of the European League of Institutes of the Arts (ELIA).

Anke Eckardt Anke Eckardt’s work encompasses sculptures/installations on an architectural scale, academic research and teaching. She has developed a series of artworks on the theme of SONIC SPACES focusing on architectural elements. She mainly works with sound (including ultrasound and infrasound) and light in combination with other materials such as water and concrete, pneumatics and motors. In 2014 she published a monograph on SONIC SPACES at Revolver Publishing, Berlin. She has received grants and prizes; her artistic work is presented internationally (Ars Electronica, NEMO, Steirischer Herbst, Club Transmediale, EMAF). Besides her solo work, Eckardt is extending collaborative projects e.g., with choreographer Shannon Cooney and artist and researcher Toby Heys, light artist Ulf Pedersen, and sculptor Sandra Peters. Since 2014 Eckardt has been Artistic Associate at the Department of Visual Arts & Architecture within the School of Architecture, Urban Planning and Landscape Design at the University of Kassel, Germany. There she also undertakes her research on sound and architecture as part of the transdisciplinary platform Building Art Invention. www.ankeeckardt.org

Katharina Gsöllpointner Katharina Gsöllpointner is a media art theoretician with a special focus on media aesthetics and the cybernetics of art. She has taught media and art studies at international universities and currently works as a researcher at the Dept. of Digital Art at the University of Applied Arts Vienna. Since the 1990s Gsöllpointner has conceived, led and carried out a series of inter- and transdisciplinary research projects on issues of media aesthetics and media arts, e.g., AESTHETIC KNOW-HOW. Language – Technology – Media (2007– 2009) or Digital Synesthesia (2013– 2016). From 1991 to 1995 she was manager of the Ars Electronica Festival in Linz (w / Peter

186

Weibel). Since the 1980s she has published extensively about media and digital art history, media aesthetics and on the transdisciplinarity of art and sciences. In her habilitation treatise Intermedia Production and Multimodal Perception. On the Aesthetics of Digital Art (2015), she formulated a media theory of digital art as an aesthetic model for the multimodality of perception. www.katharinagsoellpointner.at

Kathy Rae Huffman Kathy Rae Huffman is a freelance curator, networker, writer and media art collector. She has held significant curatorial posts at the Long Beach Museum of Art, The ICA Boston, and the former Cornerhouse, Manchester (UK). She was Professor of Electronic Arts at Rensselear Polytechnic Institute, Troy, NY. For three decades, she has consulted, presented special programs, curated, juried, administered and coordinated events for numerous international media art festivals and arts organizations. Her research topics have focused on artists’ television, video art, and feminist strategies in online environments. Huffman co-founded FACES: Gender / Technology /Art, an online community for women in 1997. Huffman was awarded an MFA (summa cum laude) in Exhibition Design and Museum Studies from the School of Fine Arts, California State University, Long Beach. From 1990 – 2013, Huffman lived and worked in Vienna, Manchester and Berlin, and traveled extensively in Central and Eastern Europe. She currently resides in Southern California.

Karl Heinz Jeron Karl Heinz Jeron is a German artist living in Berlin whose practice is idea-based and focuses on the treatment of social issues. He uses customized small robots and references from his personal experience to create situations that can disrupt preconceived notions and challenge the position of the viewer. Recent exhibitions, events and collaborations include the ZKM Karlsruhe, Ars Electronica Linz, Documenta X, ICA London, Walker Art Museum Minneapolis, Berlinischen Galerie Berlin and the Museum of Modern Art San Francisco. www.jeron.org

187

Sarah Kenderdine A former maritime archaeologist, Professor Sarah Kenderdine researches at the forefront of interactive and immersive experiences for museums and galleries. In widely exhibited installation works, she has amalgamated cultural heritage with new media art practices, especially in the realms of interactive cinema, virtual and augmented reality and embodied narrative. At the University of New South Wales she is director of Expanded Perception and Interaction Centre (EPIC) and the Laboratory for Innovation in Galleries, Libraries, Archives and Museums (iGLAM). In the last 10 years she has created over 80 exhibitions including a multi-award-winning new museum in South India. She’s authored six books and later this year will complete a co-authored monograph Theorizing Digital Cultural Heritage for a Complex, Turbulent and Entangled World with MIT Press. www.niea.unsw.edu.au

kondition pluriel (Martin Kusch / Marie-Claude Poulin) Martin Kusch studied art history, philosophy and painting in Berlin, and media arts with Peter Weibel in Vienna. He is the founder and director of the Dome-lab and the project leader of the European Mobile Dome Lab for Artistic Research (E / M / D /L), at the Dept. for Digital Art, University for Applied Arts Vienna. Founder and artistic co-director of the media performance group kondition pluriel, he is particularly interested in the transformation processes of the electronic media from within performative contexts, and on how digital technologies influence our perception of the body and space. His works have been presented at numerous festivals and institutions, such as: Ars Electronica (Linz), CYNETart (Dresden), CDA (Enghien les Bains), EMPAC (Troy), FIND (Montreal), ICA (London), ISEA (Nagoya, Helsinki, Essen), SAT (Montreal), Transmediale (Berlin), Mois-Multi (Quebec City), ZKM (Karlsruhe) and Museumsquartier (Vienna). Born in Québec, Canada. Marie-Claude Poulin is a performance researcher, choreographerperformer and interdisciplinary artist who specializes in the combination of dance and new technologies. Between 1985 and 2000, she worked as a performer, notably with choreographers Benoît Lachambre and Meg Stuart / Damaged Goods. Co-founder of the media-performance group kondition pluriel (2000), she has created ground-breaking artistic experiences and performances using a wide range of cutting-edge tech tools at the intersection of dance, somatic practices and interactive installations. Marie-Claude Poulin has presented performances and installations, participated in symposia and conferences and conducted creative workshops around the world. Since 2013 she has been a lecturer at the University of Applied Arts Vienna, where she collaborates on international research projects such as E / M / D/ L and Digital Synesthesia. www.konditionpluriel.org / www.emdl.eu

AUTHORS’ BIOGRAPHIES

188

Patricia Köstring Patricia Köstring is a Vienna based publicist and cultural manager. In the 1990s she ran the Köstring/Maier gallery in Munich, an artspace dedicated to contemporary minimalism and site-specific artistic practice (together with Stephan Maier). Since she moved to Austria in 1998 she has been working in the fields of contemporary art as well as cultural politics. Currently she is a member of the editorial board of the cultural politics reader kamion and Senior Artist at the Dept. of Digital Art at the University of Applied Arts Vienna.

Alan Kwan Alan Kwan is a digital media artist and researcher. His overarching interest is in cross-disciplinary understandings of cinema and performance as well as new forms of artistic language and storytelling enabled by virtual reality and flying machines. He has presented widely as a speaker and artist-in-residence. His projects have been shown at exhibitions around the world, including the Ars Electronica Festival in Austria, ZKM Centre for Art and Media in Germany, and Museum of Contemporary Art (MOCA) Shanghai, and were also featured in major media including Discovery Channel, WIRED, Popular Science and Boston Globe. In 2014, he received the Hong Kong Arts Development Council Award for Young Artist (Media Art), and the Asian Cultural Council Fellowship to pursue his master’s degree at the Massachusetts Institute of Technology (MIT), where in 2015 he was awarded first place in the MIT Harold and Arlene Schnitzer Prize for his body of work combining art and technology. wwww.kwanalan.com

Karen Lancel / Hermen Maat Artists Lancel and Maat create meeting places in city public spaces and the digital domain. Their seductive visual performances and installations include objects, video, prints, drawings, networked databases, projections and digital networks. Audience members are invited to “artistic social labs” as “co-researchers”. In carefully designed and hosted meeting rituals, they reflect on their perception of the smart city and their experience of body, presence, identity, privacy, reciprocity and trust. In their works the artists research contemporary social systems and “trust” systems. They deconstruct and alter control and self-quantifying technologies in order to facilitate new intimate meeting experiences; allowing experimental, sensory syntheses. Audience participation is digitized, translated and printed into “social portraits of living in a networked society”. Shows: Ars Electronica, Linz; ZKM, Karlsruhe; Rijksmuseum Amsterdam; Shanghai World Expo 2010, DDC / Mobile City; Venice Bienniale 2015 / Chinese Pavillion; Public Art Lab Berlin –

189

Connecting Cities; Stedelijk Museum Amsterdam; Banff Center Canada; Art Center Nabi Seoul South-Korea; Eyebeam New York; ISEA Hong Kong 2016; ISEA2011 Istanbul; ISEA04 / Kiasma Museum Helsinki; Urban Screens08 Melbourne; NIMK Amsterdam; V2_Institute Rotterdam; Transmediale Berlin; Utrecht University Play, Perform, Participate; Beijing Contemporary Art Center (BCAC); Holland Festival Amsterdam / De Balie; Frascati Theatre Amsterdam; Eye Film Institute Amsterdam; De Appel Amsterdam; TASIE-2012 TsingHua University, Science & Technology Museum Beijing; TASIE-2006 Millennium Museum Beijing. Karen Lancel is an artistic PhD candidate at Technical University of Delft: “Participatory Systems Initiative”; she was a member of the AHK Amsterdam research group “ARTI” (Artistic research, Theory & Interpretation) and headed the interactive media art department, MFA Frank Mohr Institute Groningen 2005-2008. Hermen Maat teaches media art at the Minerva Art Academy Groningen, and is a member of the research group on Image in Context. www.lancelmaat.nl

Geert Lovink Geert Lovink is a Dutch /Australian media theorist, internet critic and author of Zero Comments (2007), Networks Without a Cause (2012) and Social Media Abyss (2016). Since 2004 he has been a researcher at the School for Communication and Media Design at the Amsterdam University of Applied Sciences (HvA), where he is the coordinator of the Institute of Network Cultures (INC). His center recently organized conferences, publications and research networks such as Video Vortex (the politics and aesthetics of online video), Unlike Us (alternatives in social media), Critical Point of View (Wikipedia), Society of the Query (the culture of search), MoneyLab (bitcoins, crowdfunding & internet revenue models) and the future of art criticism project De Kunst van de Kritiek. In 2015 the INC digital publishing research was split off into the Publishing Lab. Since 2009 he has been a professor at the European Graduate School (Saas-Fee) where he supervises PhD students. www.networkcultures.org

Lawrence E. Marks After receiving a PhD in experimental psychology from Harvard University in 1965, Lawrence Marks moved to Yale University, where he is currently professor of environmental health sciences and psychology, and to the affiliated John B. Pierce Laboratory, where he is Fellow and Emeritus Director. Marks’s research deals broadly with the processes and phenomenology of human perception: In particular, Marks’s work emphasizes the study of multisensory perception – the mechanisms by which signals from multiple sensory modalities combine and interact

AUTHORS’ BIOGRAPHIES

190

to produce perceptual experiences – and in the ways that multisensory perceptual information is coded in language. For more than four decades, Marks has studied synesthesia in perception and language, including the role of synesthesia in metaphor. Marks is the author or co-author of more than 200 journal articles, book chapters, and conference proceedings, as well as two books, including The Unity of the Senses: Interrelations among the Modalities. www.jbpierce.org

Marcello Mercado Biological theories, mathematical principles, and technology are founding elements in Mercado’s work. His interdisciplinary projects create a dialogue between visual art, robotics, bio-art and music. In 1996 he was Fellow at The John D. and Catherine T. MacArthur Foundation, The Rockefeller Foundation and The Lampadia Foundation and in 1999 Fellow at The Academy of Media Arts Köln; in 2000 Post-Production Residence at the Centre International du Creation Vidéo Pierre Schaeffer, Montbéliard, France. He received the Grant for Media Art 2013 from the Foundation of Lower Saxony at the Edith-Russ-Haus for Media Art. Mercado was awarded with the Honorary Mention, Prix Ars Electronica 2012 in Interactive Art; He has exhibited in many museums, galleries and festivals, amongst others at the ZKM Karlsruhe; Biennale di Venezia; Centre Pompidou; The Taipei Fine Arts Museum; Museo Nacional Centro de Arte Reina Sofía; 11th Biennale de l’Image en Mouvement, Genéve; Kunsthaus Dresden; Museo de Arte Moderno de Buenos Aires. www.marcellomercado.com

Aleksandra Mroczko-Wąsowicz Aleksandra Mroczko-Wąsowicz is currently Associate Professor of Philosophy and Cognitive Science at the National Yang-Ming University in Taipei, Taiwan. Her research centers on philosophy of mind, philosophy of perception, philosophy of psychology and cognitive science, neurophilosophy, consciousness, and synesthesia. She received her Ph.D. in Philosophy from the University of Mainz in 2011 with her dissertation on The Unity of Consciousness and the Phenomenon of Synesthesia. Aleksandra Mroczko-Wąsowicz has published articles on synesthesia and other topics in cognitive science and philosophy of perception in the Oxford Handbook of Synesthesia, Open MIND, Consciousness and Cognition, Cortex, Frontiers in Human Neuroscience, Journal of Vision, and Frontiers in Psychology. She is an associate editor of Theoretical and Philosophical Psychology and handling editor of the Research Topic Perception-Cognition Interface & Cross-Modal Experiences: Insights into Unified Consciousness at Frontiers in Consciousness Research. www.mroczko-wasowicz.com

191

Catherine M. Mulvenna Catherine Mulvenna received an MA in experimental psychology in 2003 and an MSc in psychological research in 2004, both from the University of Glasgow; followed by an MPhil in 2006 and a PhD in 2012, both in cognitive neuroscience and psychology, from University College London. Subsequently, Mulvenna held postdoctoral positions at Yale University before returning, in 2013, to the University of Glasgow where she is now training at the Institute of Health and Wellbeing. The author or co-author of ten articles and book chapters, Mulvenna’s research has focused on the neural and psychological mechanisms that underlie synesthesia, with special interest in the connections between synesthesia and creativity. www.scn.ucla.edu/people/mulvenna.html

Danko Nikolić The main motive for my studies is the explanatory gap between the brain and the mind. My interest is in how the physical world of neuronal activity produces the mental world of perception and cognition. I am associated with the Max-Planck Institute for Brain Research, the Ernst Strüngmann Institute, the Frankfurt Institute for Advanced Studies, and the University of Zagreb. I approach the problem of explanatory gap from both sides, bottom-up and top-down. The bottom-up approach investigates brain physiology. The top-down approach investigates behavior and experiences. Each of the two approaches has led me to develop a theory: The work on behavior and experiences led me to discover the phenomenon of ideasthesia (meaning “sensing concepts”). The work on physiology resulted in the theory of practopoiesis (meaning “creation of actions”). www.danko-nikolic.com

Regine Rapp Regine Rapp is an art historian and curator. Her specific fields of research for 20th and 21st century art are installation art, image text theory, artist books, and art & science collaborations. She worked as Assistant Professor for Art History at the Burg Giebichenstein Art Academy Halle until autumn 2013. She is co-director of Art Laboratory Berlin, which she co-founded in 2006. She curated over 30 exhibitions (e.g., the series Time and Technology, and Synaesthesia) and has published several books. She conceived and realized the international Sol LeWitt_ Symposium at Art Laboratory Berlin. She developed the international transdisciplinary conference Synaesthesia. Discussing a phenomenon in the arts, humanities and (neuro-)science, 2013 at Art Laboratory Berlin. Her new publication [macro]biologies & [micro]biologies. Art and the Biological Sublime in the 21st Century theoretically reflects Art Laboratory Berlin’s last series from 2014 –15. www.artlaboratory-berlin.org

AUTHORS’ BIOGRAPHIES

192

Ulla Rauter Ulla Rauter studied Transmedia Art at the University of Applied Arts Vienna. As a media artist and musician, she works at the interface between sound and fine art – her works include performative sculptures, music performances and self-built instruments. In 2009 she received a working scholarship from the Austrian Federal Ministry for Science and Research; in 2011 a start-up scholarship from the Austrian Federal Ministry for Education, the Arts and Culture. Exhibitions (selection): Silent roomnumberOne, Mario Mauroner Contemporary Art Vienna 2015 | Erwartung, Projekt Fabrika, Moscow 2014 | Out of the Box, MAK Vienna, 2013 | MEX21, Künstlerhaus Dortmund, 2013 | EAR ZOOM Sonic Arts Festival, Ljubljana 2013 | ART, SCIENCE AND SOCIETY / 12th ELIA Biennial Conference, Vienna 2012 | on to red, Galerie der Künstler, Munich 2012 | SIGNS & SIGNALS, Ars Electronica, Linz, 2009 | Austria for Beginners, Museum of New Art, Detroit 2009 www.ullarauter.com

Christoph Reiserer Christoph Reiserer studied musicology, music education and philosophy in Munich and Berlin and took private lessons in composition with Stefan Zorzor.  He has been producing his own projects since 1998; for example up & down (2001, Deutsches Museum Munich), so far (2003 as artist-in-residence at the CCA Glasgow, UK) and fluX (2008 for the 850th anniversary of the City of Munich). His recent work increasingly incorporates more video and electronics, including the sound installation tisch (2005), U-Musik (2006, Munich subway stations) and tunnel ende (2010, Klangspuren Schwaz, Austria). He composed three chamber operas: Und wenn wir dann soweit sind, können wir anfangen (2007), Die Nacht des Brokers (2010) and President Jekyll (2012). For the Münchener Biennale he realized the concert installation some work (2010, Museum Villa Stuck).  Awards for his work include the Leipzig Improvisation Competition in 1997 and the Music Prize from the City of Munich in 2009. www.christophreiserer.net

193

Chris Salter Chris Salter is an artist, University Research Chair in New Media, Technology and the Senses at Concordia University, Co-Director of the Hexagram network for Research-Creation in Media Arts, Design, Technology and Digital Culture in Montreal and Interim Co-Director of the newly-founded Concordia Institute for Digital Art, Culture and Technology. He studied philosophy and economics at Emory University and completed a PhD in directing and dramatic criticism at Stanford University where he also researched and studied at CCMRA. His work has been seen all over the world at such venues as the Venice Architecture Biennale, Vitra Design Museum, BIAN 2014 (Montreal), LABoral, Lille 3000, CTM Berlin, National Art Museum of China, Villette Numerique, Todays Art, Mois Multi, Transmediale, EXIT Festival (among many others. He is the author of Entangled: Technology and the Transformation of Performance (MIT Press, 2010) and Alien Agency: Experimental Encounters with Art in the Making (MIT Press, 2015). He is currently working on a new collaborative book project for the MIT Press focused on the contemporary interconnections between art, theory and research. www.chrissalter.com

Ruth Schnell Ruth Schnell is a media artist based in Vienna. Since 2010 she has been Head of the Digital Art Department at the University of Applied Arts Vienna. She has been working with computer-aided tools since the mid-1980s. Her corpus of work, which includes video installations, interactive video environments and light installations, explores the nature of human perception and the relationship between human perception and the human body. She is considered an expert in dynamic projection in particular. Moreover, she has done pioneering work in programming electronic LED light sticks visualizing the after-image phenomenon in different series of works. Solo and group exhibitions include: Satosphère, Society for Arts and Technology [SAT], Montreal /CA, 2015; Moscow Biennale of Contemporary Art / RU, 2011 and 2013; ZKM / Center for Art and Media Karlsruhe / DE, 2010 and 2008; Bienal Internacional de Arte Contemporáneo de Sevilla / BIACS 3 / ES, 2009; Akademie der Künste Berlin / DE, 2004/05; California Science Center, Los Angeles / USA, 2004; Kunsthaus Bregenz / AT, 2002; mumok – Museum moderner Kunst Stiftung Ludwig Wien / AT, 2000; 46. Biennale di Venezia / Austrian Pavilion / IT, 1995. www.ruthschnell.org

Romana K. Schuler Romana K. Schuler, art historian, wrote her dissertation on experimental perception in the arts and sciences (“Seeing Motion“ 2015). The emphasis was on oscillopsia in virtual reality. She has researched, written, taught

AUTHORS’ BIOGRAPHIES

194

and curated on new media in contemporary art since 1987. She conducted research on Austrian media art from 1992 to 1994 with an emphasis on the concept of sculpture in digital media and from 2013 to 2016 within the arts-based research project “Digital Synesthesia“. From 1995 to 2005 she was involved in the development of the Leopold Museum, Vienna. Since 2010 she has lectured on the history of experimental perception in the arts and sciences at the University of Applied Arts Vienna. In 2015 she co-curated the exhibition “Contemporary Code — Artistic Research“ at the City University of Hong Kong. www.romana-schuler.at

Jeffrey Shaw Jeffrey Shaw has been a leading figure in new media art since the 1960s, and in a prolific oeuvre of widely exhibited works has pioneered the use of digital media in the fields of virtual and augmented reality, immersive visualization environments, navigable cinematic systems and interactive narrative. Shaw was the founding director of the ZKM Institute for Visual Media Karlsruhe (1991– 2002), co-founded and directed the UNSW iCinema Centre for Interactive Cinema Research (2003– 2008), and since 2009 has been Chair Professor of Media Art at City University in Hong Kong where he is also Director of the Applied Laboratory for Interactive Visualisation and Embodiment. In 2014 he was appointed Visiting Professor at both the Central Academy of Fine Art, Beijing and the Institute of Global Health Innovation, Imperial College, London. In 2015 Shaw was awarded the Ars Electronica Golden Nica for Visionary Pioneer of Media Art, and in 2016 became Honorary Professor at Danube University, Krems. www.jeffreyshawcompendium.com

Miriam Spering Miriam Spering is a psychologist and neuroscientist investigating how the brain uses visual information to control movements. She is Assistant Professor in Ophthalmology & Visual Sciences at the University of British Columbia (UBC), Vancouver, Canada, and a member of the Centre for Brain Health and Institute for Computing, Information and Cognitive Systems at UBC. She obtained her graduate degree (Diplom) in Psychology from the University of Heidelberg, Germany, in 2002. She then moved on to the University of Gießen, Germany, to work with Karl Gegenfurtner for her doctoral degree in Psychology (summa cum laude, 2006). She completed a postdoctoral fellowship, funded by the German Research Foundation, with Marisa Carrasco at New York University, New York, and joined UBC in 2011. Her research has been recognized by many awards, most recently an Early Career Scholar Award by UBC’s Peter Wall Institute for Advanced Studies, an inter-disciplinary research institute. www.visualcognition.ca/spering/people.html

195

David Strang David Strang is an artist and researcher working with sound and interactive elements. His work explores the creative potential within the movement of noise in and around systems of sound and light by building / hacking bespoke devices and tools for performance, workshop, installation and intervention. Through processes of playful interaction and making, his practice investigates the links between objects, material, consciousness and the body. He currently lives and works in the UK and is a Lecturer in Music at Plymouth University. www.davidstrang.co.uk

Tamiko Thiel Tamiko Thiel is a visual artist exploring the interplay of space, the body, cultural memory and identity. She has won awards from the MacDowell Colony, MIT, WIRED Magazine, Japan Foundation and IBM Innovation Award, and exhibited internationally at venues such as the Istanbul Biennial, Art Gwangju, Corcoran Gallery of Art, Contemporary Istanbul, ICP / NY, ZKM, Tokyo Metropolitan Museum of Photography, Ars Electronica, Zero1 Biennial, SIGGRAPH and ISEA. Her work is featured in the references Digital Art by Whitney Museum curator Christiane Paul, World of Digital Art by DAM Berlin gallerist Wolf Lieser, Not Here Not There special issue of Leonardo Electronic Almanac and The Total Work of Art: From Bayreuth to Cyberspace by Stanford professor Matthew Smith. She has taught and lectured at institutions such as Carnegie Mellon University, the MIT Media Lab, the Bauhaus-University / Weimar, the USC School of Cinema-Television, the Berlin University of the Arts and Nanyang Technological University / Singapore. www.tamikothiel.com

Vincent Van Uffelen Vincent Van Uffelen works with code, hacked devices, and custom electronics to create light and sound as a medium for his artistic enquiry into process, complexity, systems, and the fleeting moments of experience. Driven by his interest in change and the entanglement of body and mind in becoming, he focuses on the performative aspects in his practice. His work also encompasses workshops, interventions, and installations. He currently lives and works in London, UK . www.vincentvanuffelen.com

Peter Weibel Chairman and CEO of ZKM | Center for Art and Media Karlsruhe and professor of media theory at the University of Applied Arts Vienna, Peter Weibel is considered a central figure in European media art on account of his various activities as artist, theoretician, and curator. He publishes

AUTHORS’ BIOGRAPHIES

196

widely in the intersecting fields of art and science. His career has taken him from studying literature, medicine, logic, philosophy, and film in Paris and Vienna and working as an artist to head of the digital arts laboratory at the Media Department of New York University in Buffalo (1984 to 1989) and founding director of the Institute of New Media at the Städelschule in Frankfurt /Main (1989– 1995). As artistic director he was in charge of Ars Electronica in Linz (1986– 1995), Seville Biennial (BIACS3, 2008), and Moscow Biennale of Contemporary Art (2011). He commissioned the Austrian pavilions at Venice Biennale (1993– 1999) and was chief curator of the Neue Galerie Graz (1993 to 2011). www.peter-weibel.at

Siegfried Zielinski Siegfried Zielinski is professor of media theory, archaeology and variantology of the media at the Berlin University of the Arts, Michel Foucault Professor for techno-aesthetics and media archaeology at the European Graduate School Saas Fee (CH), and director of the Vilém Flusser Archive in Berlin. He was Founding Rector of the Academy of Media Arts Cologne (1994–2000) and in 2015 was elected as the new Rector of the Staatliche Hochschule für Gestaltung Karlsruhe. He is the author of numerous books and articles as well as editor of the book series Variantology: On Deep Time Relations between the Arts, Sciences, and Technologies, of which six volumes have been published so far (2005–2013). Siegfried Zielinski is a member of the North-Rhine-Westphalia Academy of Sciences, Humanities, and the Arts, the Academy of the Arts Berlin, and the Magic Lantern Society of Great Britain.

197

Imprint

198

Katharina Gsöllpointner, Ruth

Concept and editing:

Schnell, Romana K. Schuler, all:

Katharina Gsöllpointner

University of Applied Arts Vienna, Photo credits:

Vienna, Austria

© artists, authors, DIGITAL Library of Congress Cataloging-

SYNESTHESIA. Exhibition photos

in-Publication data:

by Birgit and Peter Kainz, Martin

A CIP catalog record for this book

Kusch (p. 172, 179), Johnny Ranger

has been applied for at the Library

(p. 182), Nikola Tasic (p. 182)

of Congress. All images published with the Bibliographic information

kind permission of the copyright

published by the German National

holders. Although every effort was

Library:

made to obtain proper credit infor-

The German National Library lists

mation and permission to repro-

this publication in the Deutsche

duce images, the publishers would

Nationalbibliografie; detailed

be grateful to receive information

bibliographic data are available on

from any copyright holder not

the Internet at http://dnb.dnb.de.

credited herein.

This work is subject to copyright.

Texts credits: © authors

All rights are reserved, whether the whole or part of the material is

Translation, copy editing, proof

concerned, specifically the rights

reading: Andrea Lyman, Alun

of translation, reprinting, re-use

Brown, Gretchen Simms

of illustrations, recitation, broadcasting, reproduction on micro-

Graphic Design:

films or in other ways, and storage

Grafikum – Studio für Gestaltung,

in databases. For any kind of use,

www.grafikum.com

permission of the copyright owner must be obtained.

Printing:

© 2016 Walter de Gruyter GmbH,

Wolkersdorf, Austria

Berlin/Boston

Printed on acid-free paper

Holzhausen Druck GmbH,

produced from chlorine-free pulp. TCF ∞ Printed in Austria

ISSN 1866-248X ISBN 978-3-11-045934-0 e-ISBN (PDF) 978-3-11-045993-7

The artistic research project

e-ISBN (EPUB) 978-3-11-045948-7

DIGITAL SYNESTHESIA has been

www.degruyter.com

Fund FWF (2013 –2016).

www.digitalsynesthesia.net

In cooperation with School of

funded by the Austrian Science

Creative Media / City University of Hong Kong (Prof. Jeffrey Shaw)

199