Digital art as Synesthesia Digital art as synesthesia Multidisciplinary compendium of digital synesthesia in multime
414 106 10MB
English Pages 200 Year 2016
Table of contents :
Contents
Foreword
Introduction
Part 1. Theories on Digital Synesthesia
Syn-Aesthetics of Digital Art
Synesthesia: Perception, Language, Digital Art
A Philosophical Perspective on Unified Conscious Experience in Synesthesia: Insights from Philosophy of Perception and Aesthetics
Ideasthesia and Art
Transformation, Transliteration and Translation. Synesthesia and Multisensory Perception in Contemporary Visual Art
Sensing Digital Art: Aesthetic Acts Beyond (the two) Synesthesia(s)
Digital Synesthesia: Reminiscing about Science and Art
The Incidental Synesthete
The Interaction between Seeing and Moving: Eye Movements as a Window to the Mind?
Digital Synesthesia
Part 2. Artistic Research on Digital Synesthesia
VERTICAL 2
Space Time
At Play
Diver
E.E.G. KISS
The Flying Umbrella Project
Bestiary for the Minds of the 21st Century: Genomic Opera
Sound Calligraphy
facades
MotU #4–#6
Topography of Movement
IN_SIDE VIEW
Transmission+Interference
I am Sound
Data Music
Part 3. Digital Synesthesia Exhibition
Angewandte Innovation Laboratory, Vienna, 2016
Authors' Biographies
Imprint
Digital Synesthesia
Edition Angewandte Book Series of the University of Applied Arts Vienna Edited by Gerald Bast, Rector
Digital Synesthesia
Katharina Gsöllpointner Ruth Schnell Romana K. Schuler (Eds.)
A Model for the Aesthetics of Digital Art
Contents 06 – 07 Foreword
Gerald Bast 08 – 09 Introduction
Katharina Gsöllpointner, Ruth Schnell, Romana K. Schuler
Part 1
11 – 28 Katharina Gsöllpointner
Theories on Digital Synesthesia
Syn-Aesthetics of Digital Art 29 – 34 L awrence E. Marks / Catherine M. Mulvenna
Synesthesia: Perception, Language, Digital Art 35 – 40 Aleksandra Mroczko-Wąsowicz
A Philosophical Perspective on Unified Conscious Experience in Synesthesia: Insights from Philosophy of Perception and Aesthetics 41 – 52 D anko Nikolić
Ideasthesia and Art 53 – 58 Regine Rapp
Transformation, Transliteration and Translation. Synesthesia and Multisensory Perception in Contemporary Visual Art 59 – 64 C hris Salter
Sensing Digital Art: Aesthetic Acts Beyond (the two) Synesthesia(s) 65 – 80 Romana K. Schuler
Digital Synesthesia: Reminiscing about Science and Art 81 – 84 J effrey Shaw/Sarah Kenderdine
The Incidental Synesthete 85 – 90 M iriam Spering
The Interaction between Seeing and Moving: Eye Movements as a Window to the Mind? 91 – 98 Peter Weibel
Digital Synesthesia
Part 2
Artistic Research on Digital Synesthesia
102 – 105 Anke Eckardt
VERTICAL 2 106 – 109 Karl Heinz Jeron
Space Time 110 – 113 kondition pluriel
At Play 114 – 117 kondition pluriel
Diver 118 – 123 Karen Lancel / Hermen Maat
E.E.G. KISS 124 – 125 A lan Kwan
The Flying Umbrella Project 126 – 131 Marcello Mercado
Bestiary for the Minds of the 21 st Century: Genomic Opera 132 – 135 U lla Rauter
Sound Calligraphy 136 – 139 R uth Schnell
facades 140 – 143 R uth Schnell
MotU #4–#6 144 – 147 R uth Schnell
Topography of Movement 148 – 149 J effrey Shaw/Sarah Kenderdine
IN_SIDE VIEW 150 – 153 David Strang / Vincent Van Uffelen
Transmission+Interference 154 – 157 Tamiko Thiel / Christoph Reiserer
I am Sound 158 – 162 Peter Weibel
Data Music
Part 3
Digital Synesthesia Exhibition
164 – 183 Angewandte Innovation Laboratory,
Vienna, 2016
186 – 197 Authors’ Biographies 198 – 199 Imprint
Foreword
6
Gerald Bast
In his essay “The Future of The Image” 1 Jacques Ranciére poses the question, “What is being spoken about and what precisely are we being told, when it is said that there is no longer reality but only images? Or, conversely, that there are no more images but only a reality incessantly representing itself?” The images that we encounter everywhere today are mostly digital realities, projections of binary codes. The codes represent only themselves; reality is reduced to the combination of minimalisms. However, the resulting complexity of the world we experience appears to contradict this. Digitally created realities are often referred to as “soulless” and this criticism is based on a lack of complexity. Nonetheless, “The word is only made flesh through a narrative. An additional operation is always required to transform,” argues Ranciére. The same can be said about images. The image is the original form of the word, the spoken or written word referring to facts. “The logical picture of the facts is the thought,” proposes Wittgenstein in his tractatus logico philosophicus. When the picture of reality is however a result of a transformative operation, then the criticism of “soullessness” is not actually directed at the picture itself, but rather at the operation involved in creating the picture, the superposing of a narrative onto the resulting combination of binary codes. It is not the images that we produce of reality – or rather those that are imprinted on us – that are lacking in complexity, but the accompanying narratives. The one-dimensionality of thoughts, images and verbal arguments that impinges on us everywhere and which prompts us to despair at the complexity of (social and economic) facts is not owed to digitalization, but rather indicative of our insufficient ability to merge various aspects of perception. Consequently, generating complexity is a synesthetizising process. Awareness of the effects and opportunities of synesthetizising processes in a digital world characterized by both overwhelming complexity and one-dimensionality can best be raised – as is so often the case – through arts-based research.
1 Original: Le destin des images (2003)
This makes the contributions of the artists involved in the artistic research project DIGITAL SYNESTHESIA immeasurably valuable.
7
Introduction
The articles in this publication represent the artistic and scientific outcomes of the arts-based research project DIGITAL SYNESTHESIA, which was conducted from 2013 to 2016 at the Department of Digital Art at the University of Applied Arts Vienna. This project, which had the aim of researching the synesthetic capabilities of digital art, was the result of a collaboration of media artists, media and art scholars as well as neuroand cognitive scientists. The starting point of the inter- and transdisciplinary project was the observation of new individual and social modes of perception which appear to be linked to the culture of the digital world. Cultural, and therefore social and individual, modes of perception per se are always historically linked to new communications technologies and media. In the context of this work, the question is raised of how the multimedia aesthetics of digital technologies is changing human perception and how digital art can explore, represent and reflect on these changes. The thesis that digital art can — mainly due to the binary code that distinguishes it from all other arts — produce synesthetic perceptual experiences in the users marked the beginning of the project. On account of its binary code basis, the direct binary translation and transformation of various forms of one medium into another — such as sound into image or text into kinetics — is only possible with digital art. Over the course of the research process, however, the multimodal perception qualities of genuine multimedia-authored digital art have increasingly come to the foreground. The focus has shifted, so to speak, from the digital to the analog nature of digital art, in which ‘analog’ is understood as a supplementary term to ‘digital’ as opposed to a contrasting one. The ‘analog’ characteristics of digital art are above all evident in the simultaneity and entwinement of visual, auditory, olfactory, gustatory, haptic, tactile and kinesthetic media, making syn-aesthesia (a combination of several levels and areas of perception) possible at the perception level. This description of the aesthetics of digital art, focusing now far more on the nature of algorithms, or on the computer program, showed a consensus with current neuro- and cognitive scientific research on synesthesia, which in the sciences as well is no longer defined as merely cross-modal, but also as an intermodal perceptual phenomenon that is even shaped by semantic and social parameters. The research processes of the 17 international media artists involved in DIGITAL SYNESTHESIA evidently present themselves as the artworks created by these artists in the course of the project. The 15 art projects in total are described in this book from the perspective of the artists in terms of their aesthetic and communicative concepts and in relation to the theme of DIGITAL SYNESTHESIA. To highlight the theme of the research area, “primary sensory modalities”, which are supposed to reflect the diversity of the synesthetic perceptual qualities of the projects, are also assigned to each artistic project.
8
A separate section also documents the exhibition of the artworks on display at the Angewandte Innovation Laboratory in Vienna in the spring of 2016. For an overview of the current situation in neuroscientific, art- and media-theoretical research on the topic of synesthesia and digital art, scientists and theorists were invited to present the state of the art by showcasing examples of their respective findings. These articles furthermore represent the proceedings from the DIGITAL SYNESTHESIA international symposium, which was held at the University of Applied Arts Vienna in April 2016. The special aesthetic of the book cover is a reference to the multimodality of synesthesia. The words that appear on the cover are a compilation taken from DIGITAL SYNESTHESIA artwork titles and from the keywords of the theoretical articles. The terms on the inside of the cover are a collection of the senses, sensory modalities and sensory domains, and of the classifications of currently known types of synesthesia. The multimodal and multimedia aesthetic of the cover thus also represents the sensory interpretation of the findings of the artistic research project DIGITAL SYNESTHESIA. Acknowledgements We want to thank the artists of the DIGITAL SYNESTHESIA project for the production of their exciting artworks as research projects, and the scientists and scholars for the contribution of their valuable research texts. We’d also like to thank the curators who, with their media art expertise, supported the international DIGITAL SYNESTHESIA Call in 2014: Inke Arns, Charlotte Bank, Lee Ann Biddle, Josephine Bosma, Christa Blümlinger, Timothy Druckrey, Severin Dünser, Kathy Rae Huffman, Erkki Huhtamo, Geert Lovink, Ludwig Seyfarth, Mitchell Whitelaw. We are especially grateful to Jeffrey Shaw and Peter Weibel for their highly valuable support as DIGITAL SYNESTHESIA cooperation partners and as members of the DIGITAL SYNESTHESIA curatorial board. Both their extraordinary expertise in the field of media art and their invaluable advice for the selection of the participating artists has been of great importance to the success of the project.
Katharina Gsöllpointner Ruth Schnell Romana K. Schuler
9
THEORIES ON DIGITAL SYNESTHESIA Part 1
This book chapter combines ten theor which have been written by neuro- an media and art theorists. The articles re between media aesthetics, synesthetic modalities, theories of embodiment, d issues in the context of DIGITAL SYNE of the international DIGITAL SYNESTH University of Applied Arts Vienna on A
10
Syn-Aesthetics of Digital Art
Katharina Gsöllpointner
Keywords embodied perception, synesthesia, media aesthetics, digital art, kinesthesia, cross-modality, sensory domains, ideasthesia, language, metaphor
retical texts on DIGITAL SYNESTHESIA nd cognitive scientists as well as by efer to topics regarding the relationship c perception, brain activities, sensory digital technologies, and philosophical ESTHESIA. They are also the proceedings HESIA symposium which was held at the April 4 and 5, 2016. To arrive at a complete picture of synesthetic mental Image-schemas are general topological and oriental processes it is important to be aware that synesthetic
structures that are kinaesthetic in nature. They have an
metaphors are the result of verbalized synesthetic ex-
analog rather than digital character.
periences, but that in synesthetic cognition, language
(George Lakoff 1987)
is not the only modus operandi. Although synesthetic poetry and synesthetic painting have been described
By the very change of visual sensation, which we
in detail, attention to synesthetic concepts in the form
register when we move the locality of our body, the
of mediated visual, auditory or audiovisual imagery
notion of depth of space emerges. If we are unable to
has been rare. All new technologies — light, film,
move, there would be no motor system. If we were not
electronic sounds, images, algorithm — have inspired
able to control the movement of our motor system, we
artists to compose new relations between the arts and
would not be able to perceive the sensation, or con 12
to evoke new synesthetic experiences in an audience.
Optatibeatespace. lamet optiis dolor quossequi dolum struct a three-dimensional We allaudi experience
(Frans Evers 2012)
this phenomenon. Poincareé presented us with a little
quos dolupta ecaeria
philosophical curlyque and said, “My friends you need motion in order to perceive.” (Heinz von Foerster 1977)
11
Digital Art and Synesthetic Perception Synesthesia is a striking example of the brain’s ability
synesthetic perception of digital artworks? How do
to cross-modally integrate two or more sensory
media influence the sensations of the recipients and
modalities, dealing with phenomena such as sounds
users? How are the sensory modalities of one medium
being perceived as strong visual images, or even
(e.g., light and dark in image, silent and loud in sound,
objects like chairs being tasted. For a long time
bitter and sweet in taste, fresh and fetid in smell, soft
synesthesia was considered to be an “anormal” mani-
and hard in haptics/tactiles) transferred to another?
festation of perception, but during recent decades
Is there something other than the five classic sensory
it has seemed more and more apparent that
domains (vision, audio, taste, smell, touch) involved
synesthesia must be a phenomenon which is only
in the construction of synesthetic perception? Can
a concentration and an intensification of an
synesthesia-inducing stimuli be detected by means
explore the capabilities of digital art to create translational and cross-modal sensory experiences
of the various media forms of an artwork? Are certain media features especially appropriate for evoking synesthetic experiences? On the side of the aesthetic appearance of art-
and to provide synesthetic experiences for non-
works it is the artistic media, their modalities, and
synesthetes. These presumptive capabilities are
their forms which provide information about the
based on the unique and special technological and
interaction and the intertwinedness of perception
media conditions of digital art, namely on its binary
properties which induce synesthetic experiences.
code which not only allows for the direct and immedi-
These media forms manifest themselves within various
ate translation of outputs in one sensory domain (e.g.,
parameters, which I will discuss further below. Here
pixel for visual representation) into outputs in another
it is important to stress the fact that the artistic act per
(e.g., sound units for auditory representation), but
se is to bring the media into forms, i.e., to create an
above all allows for the programming of algorithms.
artwork by means of what I call aesthetic decisions.
The programming represents the artistic and aesthetic
As I have shown in earlier research contexts, artists
basis for those qualities of digital art which differen-
have an expertise about aesthetics. They have a
tiate it from all other art (including analog electronic
know-how in regard to aesthetic decisions which,
media art, video and film, digital video and film
amongst other things, includes expectations about
and their postproduction). Only the code enables
effects and impacts on the perception of their work
man-machine-interaction, machine-machine-
(Gsöllpointner and Moser 2009). Permanently making
interaction, and the machine networks, all three
these aesthetic decisions about media and their
of which are pre-conditions for digital art’s strong
forms, artists therefore develop an expertise in terms
spatiotemporal constructedness. It was a main presumption of the project that
of aesthetics; they gain an Aesthetic Know-how which is not only theoretical but above all practical, based
inter- and transdisciplinarity between artists, scholars
on the experience of conceiving, producing and
and scientists must be the essential method for ex-
manufacturing artworks. The aesthetics of digital
ploring the fascinating field of synesthesia from the
artworks concede the recipient/user of the artwork
perspective of artistic research. The mutual exchange
a much higher degree of freedom to deal with than
of knowledge between art, humanities and sciences
do non-digital artworks. It is not for nothing that the
opened new perspectives within the respective fields.
terms “interactive” (e.g., in Peter Weibel’s installation
For media studies, and here especially for media
Data Music [Fig. 1] and in Tamiko Thiel & Christoph
aesthetics, the role of the media in the exploration
Reiserers’ work I am Sound [Fig. 2] ), “immersive” (e.g.,
of synesthesia is of central interest. How exactly do
in Jeffrey Shaw & Sarah Kenderdine’s IN_SIDE VIEW
media (in their respective forms) contribute to the
[Fig. 3] ) or “responsive” (e.g., in kondition pluriel’s At
THEORIES ON DIGITAL SYNESTHESIA
Katharina Gsöllpointner
The aim of DIGITAL SYNESTHESIA has been to
12
Syn-Aesthetics of Digital Art
otherwise “normal” perceptive processing.
Play [Fig. 4] ) are fundamental to the description of
cross-, multi- and trans-modal character in this
artistic media in artworks. Digital artworks can there-
model is on the one hand determined by the sensory
fore function as best practice examples of the process
constitution of perception, but also on the other by
of perception per se, not just because of their “beau-
neuroscientific findings about synesthetic processing
ty” or other psychological features, but because they
which have revealed a strong involvement of
offer the possibility of observing “pure” forms and the
conceptual parameters in perception as a holistic
media of the latter.
bodily process. Thus, the traditional differentiation
To explore the media functions in “syn-aesthetic”
between “strong” and “weak” synesthesia (Martino
digital art, I formed a model to compare the multi-
and Marks 2001, Marks and Mulvenna 2013) might
media features of digital artworks with the multi-
have become obsolete as research has revealed
modal features in types of synesthesia by describing
new knowledge about the involvement of what
their syn-aesthetic correlations (Gsöllpointner 2015).
scientists call “semantic inducers” in synesthetic
I have suggested that digital art and synesthesia
experiences. Semantic inducers are non-sensory
can form ideal correlatives to explore their media or
spatial, temporal, and motional concepts, such as for
modal constitutions respectively by means of
instance letters, numbers, calendar units, and even
comparing the features of an artwork – such as its
swimming styles. They do not count as sensory
title, its primary and secondary artistic media, its
inducers but can still evoke synesthetic experiences
cross-modal aesthetics, or its possible semantic
(cf. Amin et al. 2011, Hale et al. 2014, Mroczko-
with correlating features of synesthesia associations –
Wąsowicz and Nikolić 2014).
like the type’s name, its primary and secondary
In the context of the multimedia conditions
sensory domains, its cross-modal effects, or its
of digital art, these semantic inducers could also be
possible semantic inducers. Fig. 5 shows how
seen as representative of a “forgotten” domain of
modal components of synesthesia can be matched
the senses. Such a “sixth sense” (Berthoz 2000,
with media components of digital artworks in terms
Fingerhut 2011) not only extends the area of the
of aesthetic correlations and therefore give
traditional five human senses but could even be
information about how forms of media contribute
considered as encompassing and integrating all
to the understanding of perceptive phenomena.
physical perceptive and cognitive processes by its
The connectedness between digital media, their forms in artworks, synesthetic perception and its
literally embodied structure: the kinesthetic domain or kinesthesia.
Synesthesia and Kinesthetics Kinesthesia as a unique sensory domain has been
Webster Dictionary 2015). However, in science there
rediscovered by the sciences in recent years, although
still lacks a consistent and standardized definition
its existence has been under investigation since as
of the term.
early as the 19th century. The exact definition of kines-
Historically, the term kinesthesia derives from early
thesia (the kinesthetic sense) is still controversial and
studies about the physiological conditions of humans
therefore varies in literature. In general, the following
and animals moving their limbs and staying in balance.
very brief description provides a good conclusion
It has also been called the “sense of locomotion” (J. C.
of the basic points: one can say that kinesthesia is
Scaliger 1557) or the “muscular sense” (C. Bell 1826).
“a sense mediated by receptors located in muscles,
H. C. Bastian proposed the term kinesthesia in 1880,
tendons, and joints and stimulated by bodily move-
and in 1906 C. S. Sherrington introduced the terms
ments and tensions”; but it is also at the same time the
proprioception, interoception and exteroception as
“sensory experience derived from this sense.” (Merriam
specifications of the “locality” of the input of sensory
13
Syn-Aesthetics of Digital Art
Fig. 2
THEORIES ON DIGITAL SYNESTHESIA Katharina Gsöllpointner
Fig. 1
Fig. 4
14
Fig. 3
Structural Correlations between Synesthesia and Digital Art
Fig. 1 Peter Weibel: Data Music, 2016 (Exhibition view AIL Vienna) Photo: Birgit and Peter Kainz
Fig. 2 Tamiko Thiel / Christoph Reiserer: I am Sound, 2016 (Exhibition view AIL Vienna) Photo: Birgit and Peter Kainz
Fig. 3 Jeffrey Shaw / Sarah Kenderdine: IN_SIDE VIEW, 2016 (Exhibition view AIL Vienna) Photo: Birgit and Peter Kainz
Fig. 4 kondition pluriel: At Play / Enjeux (Performance at Theatre La Chapelle, Montreal. Photo: Bruno Colpron, © kondition pluriel, 2015)
Fig. 5 Syn-aesthetic correlations between synesthesia and digital art (Gsöllpointner 2015)
Synesthesia (Neuroaesthetic components)
Digital Art (Aesthetic components)
Type
Title
Synesthetic effect(s) (Phenomena)
Aesthetic effect(s) (Description)
Involved primary sensory domains (e.g., auditive, visual, kinesthetic, etc.)
Involved primary (artistic) media (e.g., sound, image, spatial installation, etc.)
Secondary sensory domains
Secondary (artistic) media
Main directionality from stimulus inducer to (concurrent) sensory evocation (Additionally bidirectionality has been proven also, e.g., by Ward et al. 2008, 1295)
Main directionality from one medium to another (Bi- and multidirectionality or simultaneity is often a matter of course; intermedia)
Multimodal aspects (Involvement of many / all sensory modalities)
Multimedia aspects (e.g., media embodiment, “Gesamtkunstwerk”, etc.)
Cross-modal effect(s) (A stimulus from one sensory domain is experienced in several sensory domains)
Cross-modal aesthetic(s) (Simultanous, computerized translation from one medium into one or many others, by means of binary code)
Intermodal quality correlations of inducers and concurrents (e.g., auditive qualities like pitch correlate with visual qualities like brightness)
Intermodal quality correlations of different media (e.g., metaphorical comparisons like “sound is light”, language metaphors, etc.)
Applicable exemplary semantic inducer(s) (e.g., personification of inducers, mirror touch inducers, etc.)
Possible semantic associations (Semantic field)
Involved spatiotemporal concepts (e.g., movement of concurrents, time units, numbers, acceleration and speed, etc.)
Spatiotemporal components (e.g., [moving] pictures, installations which require the movement of recipients, time-based artworks like film, video, interactive installations, net.art, music, game art, etc.)
Fig. 5
15
stimuli. Today it has been proved that on the physio-
system consists not only of immediate skin contact
logical level, muscular nervous activities are definitely
and tactility but also internal bodily, or somatic,
involved with the kinesthetic sense in manifold ways
sensations. Proprioception constitutes one of these
(Proske and Gandevia 2009).
somatic sensations within the haptic system. In addition to proprioception, these somatic sensations comprise kinesthesia (the sense of movement) and the
can be described, Fulkerson has suggested the
vestibular system (the sense of balance).” (Storks 2016)
model of “sensory pluralism” (Fulkerson 2014). Instead
In this model the kinesthetic sense would — together
of the dichotomy of a (conservative) concept of
with the vestibular sense (equilibrioception or the
“sensory monism” which represents a model deter-
sense of balance) — be part of the proprioceptive sense
mined by single, distinguished sensory domains on
which, as mentioned above, is connected strongly to
the one hand, and the complete elimination of the
the haptic or tactile sense. Moreover, kinesthesia also has a special, strong connection with the visual sense. Berthoz explains
that rejects any single, unified account of sensory
this interdependency of vision and kinesthetics in
modalities and their interactions, instead embracing a
reference to Merleau-Ponty’s expression “La vision est
multitude of such accounts.” (Fulkerson 2014). Already
palpation par le regard” (“Seeing is touching with the
Gibson has suggested that the senses are a “perceptual
gaze.” Translation into English by the author) (Berthoz
system” rather than a conglomeration of single senses,
2014). Spering and Carrasco, for instance, have shown
and he referred to kinesthetics as to “cut across the
how tiny eye movements (saccades) are a necessary
functional perceptual system” (Gibson 1966, cit. in
pre-condition for visual perception per se (Spering &
Fingerhut 2011). The concept of kinesthesia as an inte-
Carrasco 2015). And in the context of art perception,
grating sense would fit well into these considerations.
O’Regan and Noë also explored the sensorimotor
In general it is clear that, in addition to the classic five so-called exteroceptive senses – seeing (visual), hearing (auditory), tasting (gustatory), smelling
account of vision and visual consciousness early on (O’Regan and Noë 2001). Very often though, the term proprioception is used
(olfactory) and touching (tactile/haptic) – humans
instead of kinesthesia, and is described as the “sense
have a range of other sensory perception capabilities,
of position and posture, movement and velocity of
which are in general subsumed under the concept of
the body and body parts.” (Storks 2016) Storks con-
interoception, which again contains the so-called
cludes the manifold approaches to the exploration of
proprioception, equilibrioception, visceroception,
kinesthesia and the subsequent confusion, ambiguity
thermoception, and nociception. In this model, visceroception means the perception
and almost nebulosity of the meaning of the concept of kinesthesia in science as follows: “Paterson … con-
of internal organs like hearing the blood flow through
siders kinesthesia and proprioception to be equal
the veins, feeling one’s heartbeat or sensing one’s di-
components of somatic sensations and notes that they
gestion. Thermoception is the sensing of heat and cold,
work in conjunction with the vestibular system … Ham-
and nociception means sensing pain, both
ilton points out that proprioception comprises kines-
of which are sensed by the same receptors in the
thesia as well as the knowledge of fatigue and warmth
body (Proske and Gandevia 2009). Both of the latter
and cold, the vestibular system, the visceral sense
are also considered to be very closely attached to
and visual proprioception (the kinesthetic function of
the haptic sense of touch. However, this is also
vision) … Berthoz, in turn, classifies proprioception as
claimed for the term proprioception: “Proprioception …
a component of kinesthesia which also includes the
originally stems from the fragmentation of the sense
vestibular system and vision …“ (Storks 2016).
of touch. As such, proprioception is still classified as a component of touch or the haptic system. The haptic
THEORIES ON DIGITAL SYNESTHESIA
Katharina Gsöllpointner
concept of distinguishable senses as “eliminativism” on the other, he argues for “an intermediate view
Basically, I would like to follow Berthoz’ suggestion and subsume proprioception and the vestibular sense
16
Syn-Aesthetics of Digital Art
In the course of the ongoing academic discussions about the conception of senses and how exactly they
under the term of kinesthesia. Kinesthesia can thus
Such semantic inducers cannot be ascribed to
be simply described as “the sense of moving”
sensory stimuli origins like images, sounds, smells,
or “the moving sense.” Furthermore, this also fits in
tastes or haptics/tactiles. They are merely conceptual
with Wolve’s use of the term as distinctive to a
ideas or imaginations which induce cross-modal
concept of a dis-embodied mind: “Here, the word
perception and which are mainly temporal, spatial,
‘proprioception’ will serve as a short-hand
motional and social concepts. Semantically induced
designation for the priority of dynamic embodied
synesthesia includes, for instance, phenomena like
activity over isolated ‘mental’ and ‘physical’ regions.
mirror-touch synesthesia (Banissy and Ward 2007),
Rather than asking ‘How can a brain accomplish
or the personification of graphemes (Amin et al.
reasoning?’, the question becomes ‘How can a
2011). It has also been shown that the cognitive
brain have experiences?’, that is, ‘What is it like for
activities of the brain alone can evoke synesthetic
a brain to be embodied?’” (Wolve 1999)
experiences (Sagiv and Ward 2006). Recent neurosci-
The difficulty of defining a kinesthetic sense
entific research reports synesthetic experiences with
coherently might also be owed to the fact that it is,
non-perceptual, semantic inducers like words,
though added to the interoceptive sensory domain,
letters, numbers, time-units, musical notes, personali-
so obviously involved in all sensory experiences
ties, emotions and even moving styles of swimming.
and that it is therefore difficult to discriminate it
The swimming-style synesthesia here is of particular
from the exteroceptive sensory domains. When we
interest, as it has been shown that not only watching,
“see with the legs” (Foerster 1977), move our eyes to
but the mere imagination of people swimming in
be able to see (Spering and Carrasco 2015), touch
different styles evokes synesthetic color experiences
non-existent surfaces with our fingers (Robles-De-
in probands (Nikolić et al. 2011, Rothen et al. 2013).
La-Torre 2006), then kinesthesia is always part of
Ansorge et al. have shown for instance that the
it. It is integrating, and literally incorporating, all
“spatial meaning of an unconscious visual word,
senses in a synesthetic way, so to speak. This is
such as up, influences the perceived location of a
the case not only regarding the multi- and cross-
subsequently presented auditory target.” In a study
modal condition of synesthesia, but also regarding
they found out that unconscious processing “extends
its semantic, and even social, co-condition.
to semantic-sensory connections between different modalities” and contextualized these findings among
Spatiotemporal Concepts in Synesthetic Perception
others within the concept of embodied cognition (Ansorge et al. 2015).
As I have said before, the kinesthetic sense not only
Thus, spatiotemporal parameters of move-
integrates the other senses, but moreover – on the
ment seem to play an important role in the pro-
semantic side of perception – it also incorporates
cessing of the (syn)aesthetic perception of colors,
concepts of time and space, both of which in the
sounds, and numbers. Ward et al. have shown in
context of synesthesia research have been proven
a unique and large-scale interdisciplinary study
to be fundamentally involved in synesthetic
on synesthetic perception, that “a general tendency
perception. Mroczko-Wąsowicz and Nikolić have
for the visual experiences to move in a left-to
developed the model of ideasthesia to explain
right direction, ...” occurs (Ward et al. 2008). The
this phenomenon of the semantic origins of
study’s authors suggest that this phenomenon
synesthesia. It is “driven by semantic mechanisms
could have two possible causes: “First of all, there
as a part of mental representations whereby
is a general bias in attention to the left of space
each synesthete’s individual semantic network
that may reflect right hemispheric specialization
contains concurrents as a part of the meaning
for spatial processes ... Second, it may reflect the
of the inducing stimuli.” (Mroczko-Wąsowicz
cultural tendency for left-to-right reading.” (Ward
and Nikolić 2014)
et al. 2008)
17
Another strong cross-modality between sensory and
body schemas in language which build on spatial
spatial perception has been demonstrated by Cuskley
and motional body concepts (Lakoff and Johnson
with the example of sound and motion synesthesia:
1980/2003). Such “spatial primitive image schemas”
“… sound and motion perception are fundamentally
occur in every language in the world. They provide
connected via low-level multi-modal representations …
information about the locality and the spatiotemporal
motion perception is fundamentally multi-modal, and
context of the (speaking) body and contain spatial and
research in motion perception has shown that audi-
temporal deictic (here, there, after etc.) and directive
tory information in particular is vital to the perception
(into, up, forward etc.) indicators, container meta-
of motion” (Cuskley 2013). This strong connection
phors (in, through, out etc.), statements about the
between motion and sound in speech perception
constitution of something (part - whole), about the re-
is described by the author as a “shared cross-modal
lation of objects (in front of, faster than, etc.), the ver-
association between linguistic sound and motion”
ticality and horizontality of objects, and many more
(Cuskley 2013).
(cf. Lakoff 2015). These image schemas are genuinely
features (and of visual forms) has also been shown
an important role in every language and, moreover, in
amongst others with the well-known example of the
the context at hand, because they are the basis for the
takete- and maluma-called angular shapes
relational connections of space, time, conceptual and
(Ramachandran and Hubbard 2003); often also
social features of digital artworks, as I will show fur-
referred to as kiki and bouba, cf. for instance Danko
ther below. In deictic words the semantic meaning is
Nikolić’s article in this book. Beside spatial parameters,
fixed, but dependent on the context. Words like here
temporal parameters have also been under explo-
and there, up and down, now and tomorrow, me and
ration – for instance the circadian (or daily) rhythm
you provide information about the spatiotemporal
which seems to rely on some kind of chronoception
and social contexts of an utterance. They serve as
or “time sense” we have (Rao et al. 2001) – which
indicators of kinesthetic features in verbal expressions
indicates a strong involvement of kinesthetic parame-
but also can be expressed by non-verbal media like
ters in synesthetic experiences.
images or sounds. For instance, a sound can be
Kinesthesia therefore serves as a good example of
perceived as coming “from above” and therefore
the multimodal constitution of all sensory domains.
unconsciously have an impact on the perception of
It can be observed as overall existent across all other
visual objects (Ansorge et al. 2015).
sensory modalities – no matter if vision, audio, smell, taste, touch or the proprioceptive senses, kinesthetic
Furthermore, language can express metaphorical structures between different fields of semantic or
features of motion, direction, position, speed, rhythm,
sensory domains. The cross-modal correlations
acceleration etc. can be found in seeing, hearing,
between different sensory domains have been shown
smelling, tasting, touching and feeling. These consid-
in a number of famous studies (Marks 1974, Marks
erations are also backed by theories of language
1982, Marks 1996, Marks 1997, Yeshayahu and Gil
embodiment and embodiment of media, which basi-
2008). Metaphors are genuinely kinesthetic figures.
cally state that all perception and cognition in animals
Their fundamental structure is the transfer (the
is per se mediatized through bodily experience.
mapping). In metaphors, the features or aspects of
So, if we speak of synesthesia as the merging of
one issue/object/situation (source domain) are
the senses, then we can speak of kinesthesia as the
transferred to (or mapped onto) another issue/object/
integration of sensory, semantic and social perception.
situation (target domain). This transfer takes place from one (sensory or semantic) domain to another, as
Kinesthetics in Language
for instance the expression “bitter cold” transfers a
In the context of kinesthesia as the integrating sense,
gustatory trait to the kinesthetic (thermoceptive) do-
one must bring up Lakoff and Johnson’s theory of
main. In the expression “a piercing glance” a feature of
THEORIES ON DIGITAL SYNESTHESIA
18
Katharina Gsöllpointner
kinesthetic. Amongst them deictic expressions play
Syn-Aesthetics of Digital Art
The connectedness of vocal sound and kinesthetic
the source domain “kinesthetic” (to pierce) is
in the absence of a suitable content” (Porat and Shen
transferred onto the target domain “vision” (transfer
2015, 79). In other words: a mere form can enhance a
from the kinesthetic/haptic domain to the visual
metaphor or a metaphorical meaning respectively.
domain by means of the image schema “container”).
If we consider this to be also true for non-language
So, the kinetic aspect of transfer is inherent to a
media such as image, sound, taste, smell, haptics, and
metaphor. But metaphorical transfers of domain
kinesthetics such modal features of other media forms
features also take place in non-verbal domains, such
must be detectable by describing and categorizing
as in vision, audio and the other sensory domains.
them and investigating their metaphorical possibilities. These could for instance be kinesthetic metaphors in
Media Forms of “Imposed Metaphoricity”
all sensory domains.
It is not only metaphors themselves that are kin-
highly constrained, and is restricted almost exclusively
Although the “use of MIF in natural language is
esthetic figures, however. In language, for instance,
to metaphorical contexts”, (Porat and Shen 2015) it
forms can even be found which merely impose
would be of great interest to match these results onto
metaphorical transfers. Porat and Shen have intro-
non-verbal media in the sense that certain media
duced such a concept of “Imposed Metaphoricity” in
forms in digital artworks (MIF language forms in Porat
language (Porat and Shen 2015) to provide a model
and Shen’s study) might basically impose cross-modal
for a phenomenon “in which metaphorical processing
visual, auditory, gustatory, olfactory, haptic/tactile,
is imposed on a given expression regardless of its
and kinesthetic metaphorical processing as well. From
semantic content” (Porat and Shen 2015). They
this subsumption questions like the following arise:
suggest that besides standard language, metaphorical
How exactly do non-verbal MIF in various sensory
forms like nominal metaphors (X is Y) and similes
domains (vision, audio, taste, smell, touch, kines-
(X is like Y), so-called “metaphoricity-inducing forms”
thetics) appear and what would these aesthetics look
(MIF) are used in order to “impose metaphorical
like? How could non-verbal MIF be compared to the
processing on any two nouns, regardless of semantic
MIF in language? How could the imposing function
factors such as constituent meaning or the context of
of non-verbal MIF be described? What would visual,
the expression. This group includes various linguistic
auditory, gustatory, olfactory, haptic/tactile, and
constructions that are commonly used to express
kinesthetic MIF look like in digital art? Can semantics
intensification, such as nominal sentences with ad-
also serve as conceptual MIF? Such future research
verbial intensifiers (this X is such a Y, this X is really Y),
would provide useful information about the role of
appositive genitive constructions (this is a Y of an X),
multimodal media aesthetics in digital art in regard to
and question-like exclamations (what a Y this X is)”
synesthetic perception.
(Porat and Shen 2015). As I have stated, metaphors transfer (map) features from one semantic (or sensory) domain to another (source to target domain) to make certain characteristics of objects/issues explicit, and therefore have a kinetic/motional part within their structure. However, Porat and Shen’s findings suggest moreover that the metaphorical processing itself can be activated by two routes: the “semantic (lexical, contextual)” and the “structural (or form dependent)”. “Hence, although the process of assigning metaphorical interpretation can be described as a manipulation of semantics, it is not necessarily derived from it, and can be triggered even
19
Kinesthetics in Digital Art Digital art, due to its above mentioned algorithmic,
rhythm, which refers to the chronoceptive as well
cross- and multimedia basis, clearly has a strong
as to the auditory and visual domains.
spatiotemporal structure. Thus its aesthetics always,
Third, and this is what is interesting in the con-
and from the very beginning onward, contain funda-
text at hand, digital artworks by their technological
mental features of motion in terms of the alteration
conditions are basically compounded of kinesthetic
of space and time for an object or body. These
modalities through all sensory and semantic domains.
spatiotemporal shifts take place in the form of aes-
By the example of the 15 artworks which have been
thetic media transformations, either within the same
accomplished by artists in the course of the DIGITAL
sensory modality, or from one sensory modality to
SYNESTHESIA research process, Fig. 6 demonstrates
another. Additionally digital art — as any other art —
how modalities in various sensory domains can be
always transports a semantic field, in which contents
transferred to and from kinesthetic features of digital
of “meaning” are cached, stacked or openly narrated.
artworks.
If we consider kinesthetics as the sensory domain
basically responsible for synesthetic experiences, then
the work. Detecting verbal indicators in the titles
we must look at how kinesthetic features in media
and their linguistic function shows that 13 out of 15
forms of digital artworks accomplish this task. For this
artworks explicitly have a kinesthetic indicator in
I suggest a procedure of analyzing media forms in
their title. Only Data Music (Fig. 1) and Mirrors of the
artworks which indicate trans-modal kinesthetic
Unseen (MotU #4 – #6) (Fig. 7) do not have an explicit
features in other sensory domains as well as in the
kinesthetic indicator, although music and mirror both
“semantic field”. Before taking this step I would
strongly point to temporal and spatial aspects of
like to differentiate between three groups of art-
sound and vision.
works in which the possible analysis of kinesthetic
2. Describing the primary sensory modalities/do-
cross-modal features basically can be undertaken:
mains which are addressed by each of the artworks, it
First there is the group of artworks which are
is also obvious that in all of them the kinesthetic sense
grounded on motion media and are not necessarily
is either top-listed or among the first three modalities;
digital. In this group I count film, video, performance,
only two pieces have other senses addressed more
dance, music/sound. Second, in static, non-dynamic
obviously: although in E.E.G. KISS (Fig. 8) kinesthetics
artworks many kinesthetic modalities can be detected,
verifiably has a strong connectedness with the work’s
as for instance in paintings, sculptures or literature,
focus on touch and taste; and in Data Music (Fig. 1)
and all of them can comprise of metaphorical kines-
the playing of the “stave-strings” also has an obvious
thetic features, like a fast brush in a painting or a
kinesthetic aspect beside the strong touch aspect.
heavy piece of literature. Further spatial-kinesthetic
3. The semantic field of an artwork is of course
modalities in paintings or objects would be
dependent on the visitor’s /user’s individual and
features like right, left, high, far, wide, narrow, low,
cultural background. Nevertheless, certain formal
deep, balanced, heavy, light. Temporal-kinesthetic
indicators (such as, for instance, the title or the con-
modalities such as fast, slow, accelerating, speedy
tents of images or songs, the material that objects
might also be found in static artworks in regard to
and installations are made of etc.) provide information
the cross-modal correlations of sensory features
about the narrative field which encompasses the
(Marks 1974, Marks 1982, Marks 1996, Marks 1997,
artwork. Therefore the table gives one example of
Yeshayahu and Gil 2008). And one motional-
a possible semantic field for each of the artworks.
kinesthetic modality in static works definitely is
Most of these semantic fields also are strongly
THEORIES ON DIGITAL SYNESTHESIA
20
Katharina Gsöllpointner
1. The artwork title gives a good impression of the primary sensory domains which are addressed by
Syn-Aesthetics of Digital Art
which integrates sensory with semantic perception via its holistic bodily condition and which therefore is
1. Artwork Title Verbal Indicators of Kinesthesia
2. Primary Sensory Modalities / Domains
3. Exemplary Semantic Field (content of text if included as artistic medium)
4. Exemplary Kinesthetic Verbal Expressions and Metaphors
5. Trans-modal Media Aesthetics (sensory directionality)
At Play at (deictic preposition)
kinesthetics, spatioception (proprioception, touch, balance, audio, vision)
a playground
playing around
images are projected onto boxes; boxes become images (vision < > kinesthetics)
Bestiary for the Minds of the 21st Century: Genomic Opera 21st century (temporal indication)
kinesthetics, touch, vision, audio
multimedia hybrids
turning the page
transference of sound data into 3D-print into image data etc. (kinesthetics > audio and >vision)
Data Music —
touch, audio, vision
a musical concert
playing strings on one’s heart
sound is produced by moving/ touching the “stave-strings” (kinesthetics > audio)
Diver diving (act of body movement, associated with ‘down’)
kinesthetics, vision, audio
a dive
diving into sth.
an image turning upside down (vision > kinesthetics)
E.E.G. KISS (brain) waves (form of motion), kissing (act of tasting, touching & body movement)
taste, touch, vision, audio
a kiss
being touched by sth.
images of (brain) waves (kinesthetics > vision )
facades facade (haptic & spatial indicator)
vision, audio, kinesthetics (balance, visceroception)
street view content of the text
walking the streets
low tones of speech (kinesthetics > audio)
The Flying Umbrella Project flying (act of body movement, associated with ‘up’)
kinesthetics, touch, audio
flying umbrellas
on the fly
height equivalents in sound (a high tone) (kinesthetics > audio)
I am Sound I (deictic noun)
vision, audio, kinesthetics
a selfie content of the text
looking ahead
an image becomes the vibrating membrane of a loudspeaker (audio > vision > and kinesthetics)
IN_SIDE VIEW in (deictic preposition), side (spatial indication)
taste, touch, vision, kinesthetics
a walk through Angkor Wat
on the tip of the tongue
a tongue switch device for images (taste > kinesthetics and > vision)
MotU #4 – #6 (Mirrors of the Unseen) —
vision, kinesthetics
after-image, blind spot, synesthesia content of the text
blind spot
after-image (kinesthetics > vision)
Sound Calligraphy calligraphy (act of body movement)
vision, audio, kinesthetics
writing a text content of the text
drawing a circle
handwriting is sound (vision > audio)
Space Time space, time
kinesthetics, chronoception, audio
clocks, atomic time
time is drifting
a clock is loud (kinesthetics > audio)
Topography of Movement topography (spatial indicator), movement (motion indicator)
kinesthetics, vision, audio
a topographical space
body-scan
a light projection is scanning a wall (kinesthetics > vision)
Transmission+Interference transmission, interference (both spatiotemporal processes)
vision, audio, kinesthetics
a transmission with an interference content of the text
to mess something up
light is sound (vision > audio)
VERTICAL 2 vertical (body schema)
kinesthetics (balance, visceroception), audio, vision
an elevator
feeling up and down
high pitches of sound (kinesthetics < > audio)
Fig. 6
21
kinesthetically defined topics, such as e.g., a play-
process of sound, object, image and motion data
ground (Fig. 4), a musical concert (Fig. 1), a dive (Fig. 9),
transfer into each other and back again as its aesthetic
a hand tapping over walls (Fig. 10), flying umbrellas
basis. Such trans- and intermodal metaphors between
(Fig. 11), a walk in Angkor Wat (Fig. 3), writing calli-
kinesthesia and the other senses can also be detected
graphies (Fig. 12), the fingers of ticking clocks (Fig. 13),
by describing strictly formal parts of the artwork. For
etc. Additionally, in those artworks which use text as
instance in Diver (Fig. 9) images are turned upside
part of their conception, the content of the respective
down. Here a condition from the kinesthesia domain
text also represents a semantic field. So, for instance
(“upside down”) is projected onto the vision domain
in Ruth Schnell’s MotU #4 – #6 (Fig. 7), the words
(“image”). The image becomes kinesthetic, not only
appearing from the light sticks form a “narration” on
because it is — as any film or video — a moving picture
synesthesia, as does the text source (digital protocols
(> “turned”), but mainly because it is turned upside
of ASCII and frequency conversion) respectively in
down. Another striking and well known example
Transmission+Interference (Fig. 14) by David Strang
of the inter-modality of sensory features is the corre-
and Vincent Van Uffelen. In Ruth Schnell’s facades
lation in the perception of pitch (auditory domain)
(Fig. 15) the voice which reads out a text by Friedrich
and height (kinesthetic domain), which is used to
Kittler about the NSA creates a semantic field in
model a spatial experience by acoustic forms in the
regard to topics of secret services and interception.
artwork VERTICAL 2 (Fig. 17).
Additionally, it must be stressed that the semantic field is not only created by the text’s content alone,
This exemplary description of kinesthetic media fea-
but with the same impact by its media-reflective
tures in the DIGITAL SYNESTHESIA artworks is a first
aesthetics; in this case the whispering mode of
step toward exploring the suggested fundamental
the voice.
role of kinesthesia in the integration of sensory and
metaphors stem from the context of the semantic
by means of media aesthetics. It is evident that kin-
field of each artwork. They thus show strong kines-
esthetic features in digital art are an important part
thetic features, mainly demonstrated by either deictic
of both the conception and the perception of the
expressions (at, around, into, on top, up, down etc.),
artworks. Artworks here serve once more as best
and/or by words indicating motion (diving, walking,
practice examples of synesthesia research because
turning, playing, drawing, writing etc.). Such associa-
they are not just used as objects for the observation
tions might for instance be related to a linguistic
of physiological and psychological processes in art
metaphor like “on the tip of the tongue” in IN_SIDE
observers, but serve as research methods about per-
VIEW (Fig. 3), “on the fly” in The Flying Umbrella
ception themselves. They provide information about
Project (Fig. 11), or “playing strings on one’s heart”
the intertwinedness of media, form and perception by
in Data Music (Fig. 1); or they might be related to
means of their media aesthetics.
ambiguous language expressions like “turning the
For future detailed investigations about kinesthesia
page” in Bestiary for the Minds of the 21st Century
in digital art, a number of research questions needs
(Fig. 16), “looking ahead” in I am Sound (Fig. 2), or
to be explored further in detail: How exactly are modal
“being touched by something” in E.E.G. KISS (Fig. 8).
features in digital artworks perceived by visitors/
5. Trans-modal media aesthetics can be described
users? What do users perceive and sense exactly in
as the aesthetic media forms of the artwork which
their bodies and bodyparts, when they interact with
suggest the transfer of either kinesthetic feature into
digital artworks? What are the functional differences
another sensory domain, or vice versa. Bestiary for
between static and dynamic features in regard to the
the Minds of the 21st Century (Fig. 16), for instance,
process of perception? Do dynamic media “double”
does this in an extremely explicit way as it uses the
kinesthetic perception parameters in the visitors/
concept of the almost endless transformation
users by means of their motional traits? And: What
THEORIES ON DIGITAL SYNESTHESIA
Katharina Gsöllpointner
semantic components in (synesthetic) perception
22
Syn-Aesthetics of Digital Art
4. Exemplary kinesthetic verbal expressions and
role does social interaction among the audience
could also be considered as the “sense of empathy”.
have for the aesthetics of the work?
The ability to change perspectives — literally and metaphorically — would then only be possible
The last issue especially refers to Alain Berthoz’ statement that gestures as kinesthetic forms are
on the basis of kinesthesia (cf. Berthoz 2014).
“synchronizing the time of two individuals”
These social or “gesture” features of digital art are
(Berthoz 2014). According to this idea, kinesthesia
another form aspect for the aesthetic condition
would not only be the sense which integrates
of DIGITAL SYNESTHESIA which surely deserve
the other senses, semantics, time and space, but
more attention.
Fig. 7
Fig. 6 Kinesthetic features, media and metaphors in DIGITAL SYNESTHESIA artworks
Fig. 7 Ruth Schnell: MotU #4 – #6, 2016 (Exhibition View at AIL Vienna) Photo: Birgit and Peter Kainz
Fig. 8 Karen Lancel /Hermen Maat: E.E.G. KISS, 2016 (Performance at AIL Vienna) Photo: Birgit and Peter Kainz Fig. 8
23
Katharina Gsöllpointner
Fig. 9
Syn-Aesthetics of Digital Art
Fig. 10
Fig. 11
THEORIES ON DIGITAL SYNESTHESIA
24
Fig. 9 kondition pluriel: Diver (Sketch), 2016. © kondition pluriel
Fig. 10 Ruth Schnell: Topography of Movement, 2016 (Exhibition view AIL Vienna) Photo: Birgit and Peter Kainz
Fig. 11 Alan Kwan: The Flying Umbrella Project, 2016 (Performance at AIL Vienna) Photo: Birgit and Peter Kainz
Fig. 12 Ulla Rauter: Sound Calligraphy/ Performance, 2015-2016 Photo: Ulla Rauter Fig. 12
Fig. 13 Karl Heinz Jeron: Space Time, 2016 Photo: Karl Heinz Jeron
Fig. 13
25
Fig. 15
THEORIES ON DIGITAL SYNESTHESIA
Syn-Aesthetics of Digital Art
Katharina Gsöllpointner
Fig. 14
26
Fig. 16
Fig. 14 David Strang / Vincent Van Uffelen: Transmission+ Interference, 2016 (Exhibtion View at AIL Vienna) Photo: Birgit and Peter Kainz Fig. 15 Ruth Schnell: facades, 2016 © Ruth Schnell Photo: Johnny Ranger
Fig. 16 Marcello Mercado: Bestiary for the Minds of the 21st Century: Genomic Opera, 2016 (Exhibtion View at AIL Vienna) Photo: Birgit and Peter Kainz
Fig. 17
Fig. 17 Anke Eckardt: VERTICAL 2 (Sketch), 2016 Digital drawing: Anke Eckardt
27
Amin, Maina, et al. 2011. “Understanding Grapheme Personification: A Social Synaesthesia?” Journal of Neuropsychology 5: 255–82. Banissy, Michael J. and Jamie Ward. 2007. “Mirror-touch Synesthesia is Linked with Empathy.” Nature Neuroscience 10: 815–17. Berthoz, Alain. 2000. The Brain’s Sense of Movement. Cambridge, MA: Harvard University Press. Berthoz, Alain. 2014 “Emotional sharing with the artwork: gesture, gaze, empathy.“ International Workshop of the French National Research Agency (ANR) Project Visual Art and Emotion, May 20, 2014: Accessed January 4, 2016. https://www.youtube.com/ watch?v=fmiNzwIngY0 Cuskley, Christine. 2013. “Mappings between linguistic sound and motion.“ PJOS 5(1). 39–62. Evers, Frans. 2012. The Academy of the Senses: Synesthetics in Science, Art, and Education. The Hague: ArtScience Interfaculty Press. Fingerhut, Joerg. 2011. “Synaesthesia and Kinaesthetics. An Introduction.“ Habitus in Habitat III. 8, eds. Fingerhut J, Flach S and Söffner J. 7–19. Foerster, Heinz von and Moshe Feldenkrais. 1977. “A Conversation.” eds. Leri and Sutherland (transcription), San Fransisco. Fulkerson, Matthew. 2014. “Rethinking the senses and their interactions: the case for sensorypluralism.” Frontiers in Psychology. Gsöllpointner, Katharina. 2015. “Digital Synesthesia. The Merge of Perceiving and Conceiving.” Cyborgian Images. The Moving Image between Apparatus and Body. Yearbook of the Moving Image Studies Vol. 1, eds. Grabbe L, Rupert-Kruse P and Schmitz N. Darmstadt: Büchner Verlag. 108–133.
Hale, Joanna, et al. 2014. “Better together? The Cognitive Advantages of Synaesthesia for Time, Numbers, and Space.” Cognitive Neuropsychology. Lakoff, George. 1987. “Position Paper on Metaphor.“ ACL Anthology A Digital Archive of Research Papers in Computational Linguistics. Accessed November 22, 2015. http://www. aclweb.org/anthology/T87-1039 Lakoff, George. 2015. “How Brains Think: The Embodiment Hypothesis“. Keynote Address recorded March 14, 2015 at the inaugural, International Convention of Psychological Science in Amsterdam. Accessed December 02, 2015. https://www.youtube.com/ watch?v=WuUnMCq-ARQ Lakoff, George, and Mark Johnson. 1980/2003. Metaphors We Live By. Chicago: University of Chicago Press. Marks, Lawrence. 1974. “On Associations of Light and Sound: The Mediation of Brightness, Pitch and Loudness.” American Journal of Psychology 7: 173–88. Marks, Lawrence. 1982. “Bright sneezes and dark coughs, loud sunlight and soft moonlight.“ Journal of Experimental Psychology: Human Perception and Performance, Vol 8(2): 177–193. Marks, Lawrence. 1996. “On Perceptual Metaphors.” Metaphor and Symbolic Activity 11: 39–66. Marks, Lawrence. 1997 (1975). “On Colored-hearing Synesthesia: Crossmodal Translations of Sensory Dimensions.” Synaesthesia: Classic and Contemporary Readings, eds. Simon Baron-Cohen and Harrison John, 49–99. Cambridge, MA: Blackwell. Marks, Lawrence, and Catherine Mulvenna. 2013. “Synesthesia, at and near its borders.” Frontiers in Psychology 4: 1–4. Martino, Gail and Lawrence Marks. 2001. “Synesthesia: Strong and Weak.” Current Directions in Psychological Science 10: 61–65.
THEORIES ON DIGITAL SYNESTHESIA
Merriam Webster Dictionary. 2015. Kinesthesia. Accessed June 30, 2015. http://www.merriam-webster.com/ dictionary/kinesthesia. Mroczko-Wąsowicz, Aleksandra and Danko Nikolić. 2014. “Semantic Mechanisms May Be Responsible for Developing Synesthesia.” Frontiers in Human Neuroscience 8, 509: 1–13. Nikolić, Danko, et al. 2011. “Swimming-style Synesthesia.” Cortex 47: 874–79.
Wolve, Charles T. 1999. “Proprioception”. Journal of Neuro-Aesthetic Theory 1. Accessed January 13, 2016. http://www.artbrain.org/category/ journal-neuro-aesthetic-theory/ journal-neuro-aesthetic-theory-1/ Yeshayahu, Shen, and Gil David 2008. “Sweet Fragrances from Indonesia: A universal principle governing directionality in synaesthetic metaphors.” New Beginnings in Literary Studies. Newcastle: Cambridge Scholars Publishing, 49–71.
O’Regan, Kevin and Alva Noë. 2001. “A sensorimotor account of vision and visual consciousness.” Behavioral and Brain Sciences 24. 939 –1031. Porat, Roy, and Shen Yeshayahu. 2015. “Imposed Metaphoricity.“ Metaphor and Symbol 30: 77–94. Proske, Uwe, and Simon Gandevia. 2009. “The Kinaesthetic Senses.” The Journal of Physiology 587, 17: 4139–46. Ramachandran, V.S. and Edward M. Hubbard. 2003. “The Phenomenology of Synesthesia.” Journal of Consciousness Studies 10: 49–57. Rao, Stephen M., et al. 2001. “The Evolution of Brain Activation during Temporal Processing.” Nature Neuroscience 4: 317–23. Robles-De-La-Torre, G. 2006. “The importance of the sense of touch in virtual and real environments.“ MultiMedia, IEEE 13/3. 24-30. doi:10.1109/ MMUL.2006.69 Rothen, Nicolas, et al. 2013. “Psychophysiological Evidence for the Genuineness of Swimming-style Colour Synesthesia.” Conscious. Cogn. 22: 35–46. Sagiv, Noam, and Jamie Ward. 2006. “Crossmodal Interactions: Lessons from Synesthesia.” Progress in Brain Research 155: 259–71. Spering, Miriam, and Carrasco Marisa. 2015. “Acting without seeing: eye movements reveal visual processing without awareness.“ Trends in Neurosciences 38,4: 247–258. Ward, Jamie, et al. 2008. “The Aesthetic Appeal of Auditory-visual Synaesthetic Perceptions in People Without Synaesthesia.” Perception 37: 1285–96.
28
Katharina Gsöllpointner
Ansorge, Ulrich, et al. 2015. “Unconscious Cross-Modal Priming of Auditory Sound Localization by Visual Words.“ Journal of Experimental Psychology: Learning, Memory, and Cognition.
Gsöllpointer, Katharina and Sibylle Moser. 2009. “Sprache Beobachten.” Sprachmedien. Accessed January 19, 2016. http://www. sprachmedien.at/downloads/ sprachbeobachtung_loop.pdf.
Syn-Aesthetics of Digital Art
References
Synesthesia: Perception, Language , Digital Art
Lawrence E. Marks / Catherine M. Mulvenna
Keywords crossmodal correspondence, synesthesia, perception, language, metaphor, idiographic, nomothetic, digital art
“I was of three minds,
The Digital Synesthesia project “is a transdisciplinary
Like a tree
artistic research area which explores the (syn-)aes-
In which there are three blackbirds.”
thetics of digital art. It especially focuses on its technological, aesthetic and media conditions
(Wallace Stevens: Thirteen Ways of Looking at a Blackbird)
which make it possible to provide synesthetic experiences …” (Digital Synesthesia 2014). Central to the project is the question: How might digital art activate synesthesia in individuals who may not ordinarily experience it? To this we add a second, related question: How might digital art evoke experiences that parallel synesthetic experiences, while nevertheless differing in their origin and phenomenology? Consequently, this article explores issues we deem central to the relations among digital art, borderline synesthesia-like experiences (Marks and Mulvenna 2013) and synesthesia itself (Marks 2014a).
29
Synesthesia: Idiographic and Nomothetic Approaches We start by considering two complementary ap-
his entire tongue ...” (Luria 1968, 45) – the flavors of
proaches to the study of human behavior that can help
foods such as borscht being, of course, specific to
in understanding and sorting out the many facets of
particular cultures.
approaches focus on those individual characteristics or
what makes every synesthete’s experiences unique,
traits that make people unique. In contrast, nomothetic
so too does it encompass what synesthetes share –
approaches focus on characteristics that people share:
and thereby calls for a complementary, nomothetic
characteristics that are widespread, perhaps even
approach. Considerable work has focused on a
universal. Art, whether digital or not, emerges from an
panoply of perceptual correspondences, in both
artist’s imagination, which can creatively tap both what
synesthetes and non-synesthetes, that are universal.
is individual and what is universal.
Despite the individuality of synesthesia in its full,
Synesthesia refers to the evocation of sensations or
nuanced detail, the study of synesthesia also reveals
perceptions in a secondary domain through stimula-
general rules and regularities, unity amidst diversity
tion that typically evokes experiences only in a primary
in patterns of perception: A notable rule, first
domain. The domains may comprise different sensory
suggested more than a century ago (Bleuler and
modalities, as when sounds evoke visual shapes, or
Lehmann 1881), is the correspondence between
different domains of the same modality, as when printed
pitch and brightness across the otherwise diverse
numbers evoke colors (e.g., Mulvenna and Walsh 2005).
instances of sound-to-sight synesthesia (e.g.,
It is perhaps a truism to say that every synesthetic
Karwoski and Odbert 1938; for review, see Marks
perceiver, or synesthete – like every snowflake – is
2014a). Synesthetes may disagree, for example, on
unique; for the precise shapes, hues, and flavors that
the specific colors evoked by a piano note in C, but
a synesthete experiences, together with the precise
nevertheless agree that the color’s brightness
stimuli that trigger these experiences, are an individ-
increases as the note is played in increasingly higher
ual matter. Moreover, each synesthete’s experiences
octaves. Auditory-visual synesthesia in particular
are embedded within her or his culture and language.
reveals nomothetic principles: Synesthetic visual
Consider the Russian synesthete S., who, responding to
images commonly decrease in size and increase in
a loud, low pitched tone, “experienced a flavor like that
brightness as pitch increases, but increase in size as
of sweet and sour borscht, a sensation that gripped
loudness increases.
Crossmodal Correspondences in Non-synesthetes Overt Correspondences in Perception
na 2012). Indeed, several universal correspondences
Analogous crossmodal correspondences also charac-
revealed in auditory-visual synesthesia, including the
terize perception in non-synesthetes, to whom, for
three described above, have analogs in the perceptual
example, colors of greater brightness align with
experiences of non-synesthetes: pitch with brightness,
sounds of higher pitch (Wicker 1969; Marks 1974), as
pitch with size (inversely), and loudness with size (for
Wundt (1874) long ago noted. Further, the correspond-
reviews, see Spence 2011; Marks 2014b).
ence between pitch and brightness not only pervades Western cultures but also appears in the crossmodal
Overt Correspondences in Language
perception of the Himba, denizens of preliterate, no-
Importantly, correspondences between dimensions of
madic tribes from the deserts of northwest Namibia
different modalities or domains also emerge when the
who have little contact with Western culture (Mulven-
stimuli are words that describe or signify perceptual
THEORIES ON DIGITAL SYNESTHESIA
30
Lawrence E. Marks / Catherine M. Mulvenna
Just as the study of synesthesia encompasses
Synesthesia: Perception, Language, Digital Art
synesthesia: idiographic and nomothetic. Idiographic
experiences, rather than sensory stimuli that directly
correspondences perhaps develop through experi-
arouse the experiences. The words SOFT and LOUD
ence with statistical regularities in the world (Parise
imply dim and bright color sensations indirectly, much
2016), while others may reflect non-arbitrary, iconic
as soft and loud sounds do more directly. Similarly,
relations intrinsic to shared meanings (Dingemanse
the words BROWN and YELLOW imply low and high
et al. 2015).
pitch sensations, just as the dark color brown and the Fig. 1 characterizes the correspondence between
Covert Correspondences in Perception and Language
auditory pitch and visual brightness as revealed in the
The examples just described show how overt percep-
literal and connotative meanings of words referring
tual judgments provide simple, explicit, and direct
bright color yellow do more directly (Marks 1982).
to auditory and visual experiences. The words BLACK
evidence of crossmodal correspondence – e.g.,
and BROWN are both darker than YELLOW and WHITE
between visual color and musical pitch. In contrast,
and higher in pitch, whereas COUGH and DRUM
covert measures provide evidence of implicit pro-
NOTE are lower in pitch than SNEEZE and VIOLIN
cessing of correspondences, in both synesthetes and
NOTE, and also darker.
non-synesthetes. A conventional covert measure
With increasing age and cognitive development,
capitalizes on the so-called Stroop effect (Dyer 1973),
crossmodal correspondences become increasingly
which originally revealed how the ability to quickly
available to language. As children learn the direct,
name a printed word’s color (say, green) is impeded
denotative meanings of sensory words, for instance
when the word spells the name (say, ORANGE) of a
that COUGH and SNEEZE refer to acoustically
conflicting color.
different events, children also come to understand
Crossmodal versions of the Stroop paradigm
the indirect, connotative meanings: that COUGHS
show, for example, that people are faster and more
and SNEEZES, like coughs and sneezes, imply different
accurate in identifying a low-pitched tone presented
levels of lightness or brightness. Fig. 2 shows the
with a dark color, and faster in identifying a high-
results of a study that assessed three auditory-visual
pitched tone presented with a bright color – and,
correspondences in children varying in age from 3 to
similarly, identifying dark and bright colors in the
13 years. The study used both a sensory-perceptual
presence of low- and high-pitched sounds (Marks
task, where the children matched sounds and lights,
1987; Melara 1989). Interactions in speed and accuracy
such as tones differing in frequency (pitch) and lights
of response occur for the most part automatically,
differing in luminance (brightness), and a verbal-met-
and appear across many sensory modalities (e.g.,
aphorical task, where the children interpreted brief
vision and hearing) and dimensions (e.g., brightness
crossmodal metaphors, such as SOFT SUNLIGHT and
and pitch) (for reviews, see Marks 2004; Spence 2011).
LOUD MOONLIGHT. The results reveal systematic but distinctive increases with age in “recognizing” all
Importantly, interactions are similar when they arise from stimuli that evoke perceptual qualities,
three correspondences – pitch with brightness,
such as colors and pitches, and when they arise
loudness with brightness, and pitch with size – in
from words that represent perceptual qualities, such
both perception and in language. Perceptual per-
as WHITE and BLACK (Martino and Marks 1999;
formance exceeded verbal performance at all ages,
also Walker and Smith 1985). Perhaps crossmodal
consistent with the view that the correspondences
correspondences in perception and in language
originate in perception (some perhaps during infancy:
activate overlapping mechanisms, tapping the
Lewkowicz and Turkewitz 1980; Walker et al. 2010;
same underlying neural and mental representations
Haryu and Kajikawa 2012), and then become available
when perceptual stimuli evoke sensory qualities
to semantic representation; and semantic representa-
directly and when words evoke sensory qualities
tion may also play an important role in synesthesia
indirectly (see also Evers 2012; Mroczko-Wąsowicz
itself (Mroczko-Wąsowicz and Nikolić 2014). Some
and Nikolić 2014).
31
And what, then, may be the responses of non-
make crossmodal or crossdomain relations directly
synesthetic individuals to works of art, especially
available to spectators. With works relegated to a
including multimedia art? To the extent that works of
single medium such as music or visual art, however,
art generated through digital technologies activate
the crossmodal implications will be far less transpar-
synesthesia or crossmodal correspondence, the re-
ent to spectators, the range of possibilities far more
sulting experiences of digital synesthesia may engage
open ended – thereby posing a far greater challenge
the idiosyncratic, the universal, or both. Perceptual,
to artists’ creative capabilities.
cognitive, and emotional responses to a given digital
Universals provide touchstones for shared mean-
artwork, like responses to any artwork, depend on
ings and understanding. Multimedia works in par-
both the proclivities of the perceiver and the charac-
ticular provide an opportunity to deploy crossmodal
teristics of the art. Artists creating digital works of art,
correspondences, thereby heightening spectator
like artists working in other media, bring their unique
attention to an undercurrent of universal meanings.
histories, their weft of particulars drawn through a
At the same time, by engaging universal correspond-
warp of universals. Spectators too, when engaging
ences within their works, artists may thereby also
with works of art, bring their own tapestries of par-
sensitize spectators to the possibility of other, novel
ticulars and universals; for perception transcends
correspondences – of novel crossmodal mean-
reception. Perception, too, is a creative act.
ings. With digital art, as with other art forms, artists
Might digital artworks have a privileged capacity
may seek to avoid both the Scylla of overfamiliar
to evoke synesthesia or synesthesia-like experiences,
universality and the Charybdis of idiosyncratic
even in individuals who are not so predisposed?
unintelligibility, seasoning their works with dashes of
Conceivably. It has been reported, for example, that
both so that spectators may construe the art creatively
synesthesia can be induced in non-synesthetes
through what Coleridge called the “secondary
through hypnosis (Cohen Kadosh et al. 2009; but see
Imagination … co-existing with the conscious Will …
Anderson et al. 2014). Such unusual conditions aside,
[that] dissolves, diffuses, dissipates, in order to re-
the available evidence – both older (e.g., Howells
create” (Coleridge 1817, 296).
Lawrence E. Marks / Catherine M. Mulvenna
Implications for Digital Art: Digital Synesthesia
1944) and more recent (Bor et al 2014) – suggests that extensive, prolonged exposure to, for example, conjunctions of specific colors with specific sounds Synesthesia: Perception, Language, Digital Art
or letters may eventually produce synesthetic experience – or its simulacrum – in otherwise non-synesthetic adults. Even without the presence of overtly synesthetic perception (i.e., developmental synesthesia: Marks 2014a), however, people respond implicitly to, and hence covertly understand or recognize, many crossdomain correspondences. The advent of digital technologies has effected a sea change in artistic possibilities, including new opportunities to capitalize on covert correspondences. Through multimedia works, for example, artists can establish explicit correlations across auditory, visual, and linguistic experiences, tapping universal correspondences or implying novel, idiosyncratic ones, and can thereby
THEORIES ON DIGITAL SYNESTHESIA
32
Fig. 1 Ratings of pitch plotted against ratings of brightness, evoked by the meanings of words referring to visual and to auditory experiences. Words that refer to brighter colors metaphorically imply higher pitch, and words that refer to higher-pitched sounds metaphorically imply greater brightness. The crossmodal correspondence in language thereby mimics the crossmodal correspondence in perception (based on data from Marks 1982).
Fig. 2 The proportion of children of different ages who associated higher pitch with greater brightness, greater loudness with greater brightness, and higher pitch with smaller size, in tasks of auditory-visual perceptual choice (left panel) and analogous tasks of rating verbal, metaphorical meaning (right panel). Random performance in each task is 50 % (dashed line). In each, the capacity of children to recognize crossmodal correspondences increases with age, performance at a given age being superior in the perceptual task compared to the metaphorical, verbal task (data of Marks, Hammeal, and Bornstein 1987).
Fig. 1
-,,
'+% '"&&
'+% '"&&
0/
#("&&+% '"&&
/,
#("&&+ % '"&&
'+ *
'+ *
./
"%)+%$'(!%!+ '$#% !
,
Fig. 2
33
References Anderson, Hazel P., Anil K. Seth, Zoltan Dienes, et al. 2014. “Can grapheme-color synesthesia be induced by hypnosis?” Frontiers in Human Neuroscience 8:220. Published April 28, 2014. doi. org/10.3389/fnhum.2014.00220 Bleuler, Eugen and Karl Lehmann. 1881. Zwangmässige Lichetempfindungen durch Schall und Verwandte Erscheinungen. Leipzig: Fues’ Verlag. Bor, Daniel, Nicolas Rothen, David J. Schwartzman, et al. 2014. “Adults can be trained to acquire synesthetic experiences.” Scientific Reports 4:7089. Published November 18, 2014. doi.org/10.1038/srep07089 Cohen Kadosh, Roi, Avishai Henik, Andras Catena, et al. 2009. “Induced cross-modal synaesthetic experience without abnormal neuronal connections.” Psychological Science 20: 258–265. Coleridge, Samuel Taylor. 1817. Biographia Literaria; or Biographical Sketches of My Literary Life and Opinions, Volume 1. London: Rest Fenner. Digital Synesthesia. 2014, “ABOUT,” accessed October 14, 2015. http:// digitalsynesthesia.net/wp Dingemanse, Mark, Damián E. Blasi, Gary Lupyan, et al. 2015. “Arbitrariness, iconicity, and systematicity in language.” Trends in Cognitive Sciences 19: 603–615. Dyer, Frank N. 1973. “The Stroop phenomenon and its use in the study of perceptual, cognitive, and response processes.” Memory & Cognition 1: 106–120. Evers, Frans. 2012. The Academy of the Senses: Synesthetics in Science, Art, and Education. The Hague: ArtScience Interfaculty Press. Haryu, Etsuko and Sachiyo Kajikawa. 2012. “Are higher-frequency sounds
Howells, Thomas H. 1944. “The experimental development of color-tone synesthesia.” Journal of Experimental Psychology 34: 87–103.
Marks, Lawrence E., Robin J. Hammeal, and Marc H. Bornstein. 1987. “Perceiving similarity and comprehending metaphor.” Monographs of the Society for Research in Child Development 42: 1–91.
Karwoski, Theodore and Henry S. Odbert. 1938. “Color-music.” Psychological Monographs 50: 1–60.
Marks, Lawrence E. and Catherine M. Mulvenna. 2013. “Synesthesia, at and near its borders.” Frontiers in Psychology 4:651. Published September 26, 2013. doi.org/10.3389/fpsyg.2013.00651
Lewkowicz, David J. and Gerald Turkewitz. 1980. “Cross-modal equivalence in early infancy: Auditory-visual intensity matching.” Development Psychology 16: 597–607.
Martino, Gail and Lawrence E. Marks. 1999. “Perceptual and linguistic interactions in speeded classification: Tests of the semantic coding hypothesis.” Perception 28: 903–923.
Luria, Alexander R. 1968. The Mind of a Mnemonist. New York: Basic Books.
Melara, Robert D. 1989. “Dimensional interactions between color and pitch.” Journal of Experimental Psychology: Human Perception and Performance 15: 69–79.
Marks, Lawrence E. 1974. “On associations of light and sound: The mediation of pitch, loudness, and brightness.” American Journal of Psychology 87: 173–188. Marks, Lawrence E. 1982. “Bright sneezes and dark coughs, loud sunlight and soft moonlight.” Journal of Experimental Psychology: Human Perception and Performance 8: 177–193. Marks, Lawrence E. 1987. “On crossmodal similarity: Auditory-visual interactions in speeded discrimination.” Journal of Experimental Psychology: Human Perception and Performance 13: 384–394. Marks, Lawrence E. 2004. “Cross-modal interactions in speeded classification.” In Handbook of Multisensory Processes, edited by Gemma Calvert, Charles Spence, and Barry E. Stein, 85–106. Cambridge, MA: MIT Press. Marks, Lawrence E. 2014a. “Synesthesia: A teeming multiplicity.” In Varieties of Anomalous Experience: Examining the Scientific Evidence (Revised Edition), edited by Etzel Cardeña, Steven J. Lynn, and Stanley C. Krippner, 79–108. Washington, DC: American Psychological Association. Marks, Lawrence E. 2014b. “Weak synesthesia in the general population.” In Oxford Handbook of Synesthesia, edited by Julia Simner and Edward M. Hubbard, 761–789. Oxford: Oxford University Press.
THEORIES ON DIGITAL SYNESTHESIA
Mroczko-Wąsowicz, Aleksandra and Danko Nikolić. 2014, “Semantic mechanisms may be responsible for developing synesthesia.” Frontiers in Human Neuroscience 10:3389. Published August 19, 2014. doi.org/10.3389/fnhum.2014.00509
Wicker, Frank W. 1969. “Mapping the intersensory regions of perceptual space.” American Journal of Psychology 81: 178–188. Wundt, Wilhelm. 1874. Grundzüge der Physiologischen Psychologie. Leipzig; Engelmann.
Lawrence E. Marks / Catherine M. Mulvenna
This article is dedicated to the memory of Frans Evers (1948–2010).
brighter in color and smaller in size? Auditory –visual correspondences in 10-month-old infants.” Infant Behavior and Development 35: 727–732.
Mulvenna, Catherine M. 2012. On creativity in synaesthetes: Roles of neural connectivity, Cognitive control, and perceptual correspondence. Doctoral thesis, University College London. Mulvenna, Catherine M. and Vincent Walsh. 2005. “Synaesthesia: supernormal integration?” Trends in Cognitive Sciences 10:350–352. Parise, Cesare V. 2016. “Crosssmodal correspondences: Standing issues and experimental guidelines.” Multisensory Research 29: in press. Spence, Charles. 2011. “Crossmodal correspondences: A tutorial review.” Attention, Perception, & Psychophysics 73: 1–25. Walker, Peter, J. Gavin Bremner, Ushi Mason, et al. 2010. “Preverbal infants’ sensitivity to synaesthetic crossmodality correspondences.” Psychological Science 21: 21–25. Walker, Peter and Sylvia Smith. (1985). “Stroop interference based on the multimodal correlates of haptic size and auditory pitch.” Perception 14: 729–736.
34
Synesthesia: Perception, Language, Digital Art
Dedication
A Philosophical Perspective on Unified Conscious Experience in Synesthesia: Insights from Philosophy of Perception and Aesthetics Aleksandra Mroczko-Wąsowicz
Keywords philosophy of mind, philosophy of perception, aesthetics, synesthesia, the unity of consciousness, multisensory integration, cross-modal binding, semantics
Philosophy of perception and aesthetics are disci-
of perception as including the mentioned non-sensory
plines about experiencing the world. Philosophy of
aspects, and in consequence a liberal understanding
perception is about perceptual experiences. Aesthet-
of the philosophy of perception as a discipline going
ics is about aesthetic experiences, e.g., the experience
well beyond the restrictive characterization of per-
we have when looking at digital art works, or the
ception. In this way, philosophical questions about
experience we have when we identify with Wassily
perceptual and related non-perceptual mental
Kandinsky or any other synesthete. This paper
processes cover not only philosophy of perception
acknowledges a wide scope of perceptual experiences,
and aesthetics, but also a significant part of the phi-
i.e., not exclusively identified with the sensory stimula-
losophy of mind. Many issues in aesthetics may be
tion. Accordingly, what belongs to perception includes
considered to be topics in philosophy of perception
various cross-modal interactions, categorizing,
and philosophy of mind (Nanay 2015; Noe ¨ 2015). One
conceptualization processes, different kinds of top-
example is the question of whether aesthetic prop-
down non-sensory influences, and attention effects.
erties such as beauty, harmony, or coherence can be
This claim lays the foundation for an expansive view
perceived or only inferred from other properties. This
35
has recently become a part of the broader debate
17th and 18th centuries by philosophers John Locke
in the philosophy of perception, namely the discussion
(1689) and Gottfried Wilhelm Leibniz (1765), reporting
on whether high-level properties (i.e., properties
of a blind man for whom thinking of the color scarlet
over and above those directly transduced by the
was like hearing the sound of a trumpet. Nowadays
sensory modalities) are admissible for entry into the
we know more about this condition and its various
contents of perception and phenomenal conscious-
forms, but the phenomenon of synesthesia still poses
ness (Macpherson 2011b; Siegel 2010; Bayne 2009).
questions for philosophers, psychologists, and cogni-
These properties are meant to be abstract, generalized,
tive neuroscientists.
tioned aesthetic properties, kind properties
tions of synesthetic perception are complementary,
(recognizing that something belongs to a certain
but a clear distinction should be made between
category), numerical values, and other semantic
the involuntary psychological phenomenon of
properties representing some meaning such as a
synesthesia occurring in a minority of subjects and
number or a letter. In this paper I consider conscious
synesthesia in art involving the intentional creation of
experience in synesthetic perception to be a striking
multimedia that share simultaneously produced
example supporting the liberal thesis that contents of
features, e.g., an association between audition and
perceptual experience may include high-level prop-
vision by sharing data or computational process (Sagiv
erties; this means that synesthetes experience their
2009). Genuine synesthetes may experience color
extraordinary percepts as having high-level properties
photisms on some internal mental screen if they are
(Mroczko-Wąsowicz 2011; Mroczko-Wąsowicz and
associators. If they are projectors they may project
Nikolić 2014; Matey 2014).
concurrent colors onto the inducing stimulus, onto
We usually experience the world in a multisensory
a bodily reference frame such as their hands, or
manner. Our senses are not independent streams of
into the immediate peripersonal space within their
information. They influence each other so that respec-
reach. This is unlikely in cases of creative cross-modal
tive sensations converge into a unified experience.
interactions, except for the last option which seems
For some individuals, stimulation of one sensory or
prospective for synesthesia in digital arts.
cognitive stream may produce perceptual experience
There are numerous forms of developmental
in more than one modality. In such a “startling
synesthesia (i.e., healthy persons who have had
sensory blending” (Cytowic, 1997, p. 17) synesthetes
this condition for as long as they can remember)
have “a conscious experience of systematically
differing from each other not only in their respective
induced sensory attributes that are not experienced
inducer-concurrent pairings but also in etiology.
by most people under comparable conditions”
It is common to all these cases that the internally
(Grossenbacher and Lovelace 2001, 36). As a result,
generated synesthetic concurrent is an actual
the stimuli corresponding to the inducer and the
perceptual experience or sensory imagery, and not
experiences associated with the concurrent form an
merely a metaphor or association (Baron-Cohen
integrated percept (Mroczko-Wąsowicz and Werning
et al. 1987; Martino and Marks 2001; Macpherson
2012; Mroczko-Wąsowicz 2011).
2007; Deroy 2015). Developmental synesthesia
Although recent scientific studies examine cross-modal interactions, philosophers and artists
is inborn and consistent throughout a synesthete’s lifetime. There are, however, other kinds of
have already prepared the ground and initiated dis-
synesthetic-like experiences that may be induced
cussions on multisensory perception. Some promi-
temporarily under hypnosis, hallucinogenic drug-
nent examples include philosopher John Locke, the
influence, or acquired as more stable phenomena
mentioned synesthetic painter Wassily Kandinsky, and
after sense-organ or brain injury (Brogaard 2013;
composers Richard Wagner and Alexander Scriabin.
Luke and Terhune 2013; Deroy and Spence 2013;
The first remarks about synesthesia were made in the
Cohen Kadosh 2009).
THEORIES ON DIGITAL SYNESTHESIA
Aleksandra Mroczko-Wąsowicz
Philosophical, psychological, and artistic explora-
36
A Philosophical Perspective on Unified Conscious Experience in Synesthesia: Insights from Philosophy of Perception and Aesthetics
and cognitive in their nature. Examples are the men-
Synesthetic experiences are not just cross-modal,
used as a test case for philosophical theories of
but they also cross domains, i.e., besides the various
perception. Since synesthesia crosses modalities and
modalities of senses, these experiences may involve
domains, it is capable of challenging traditional
domains of bodily, motor and emotional states as
methods which make clear-cut distinctions individ-
well as domains of abstract, conceptually-represented
uating different sensory modalities as independent
entities like numbers or time units (Mroczko-
senses (O’Callaghan 1998; Gray 2011; Keeley 2013),
Wasowicz and Werning 2012). Cases of synesthetic
or separating various mental faculties or levels of
inducers going beyond traditionally denoted sensory
processing (Sagiv et al. 2011; Mroczko et al. 2009;
modalities have been found for activities such as read-
Mroczko-Wąsowicz and Nikolić 2013). Considering
ing musical notes, calculating, imagining, or just think-
the relationship between synesthetic and non-
ing of a stimulus (Dixon et al. 2000; Ward, Huckstep,
synesthetic perception can make us think of defini-
and Tsakanikos 2006; Simner 2007). For many forms of
tions of perceptual experience as being too narrow
synesthesia the inducers are human artifacts, various
and excessively rigid. It is not only individual
cultural constructs, e.g., letters, numbers, time units
differences that seem neglected or underestimated.
(days of the week, months in the calendar), words,
Understanding perception is often limited to sensory-
musical notes and instruments, or sports-specific
based veridical representations. However, a thorough
movements, e.g., swimming styles (Ward, Tsakanikos,
insight into perception should include a broad
and Bray 2006; Rich et al. 2005; Mann et al. 2009;
spectrum of top-down factors and their interactions
Nikolić et al. 2011). The categorization of these events/
with bottom-up features. Synesthesia, being clearly
objects is likely high-level, involving a conceptual
a cogno-sensory phenomenon, poses difficulties for
component. Recent studies suggest an important
conservative theories of perception, theories which
role of semantics in the induction of synesthesia. The
do not accept the inclusion of high-level (represen-
phenomenon seems to rely on a certain interpretation
tations of) properties in the contents of perception.
of the stimulus and the meaning that it has for the
Overall, synesthesia encourages the thought of
subject (Ward and Simner 2003; Ward and Sagiv 2007;
perception as a more diversified phenomenon than
Mroczko-Wąsowicz and Nikolić 2014; Rothen et al.
is generally assumed.
2013; Meier 2013; Chiou and Rich 2014). For example,
When we experience different sensory features at
synesthetic colors elicited by ambiguous graphemes
the same time, even though they do not belong to one
depend on the interpretation of the graphemes as
and the same object, we can still experience them to-
either a numeral or a letter (Dixon et al. 2006).
gether, i.e., as copresent or unified in our experience.
Research on synesthesia can offer profound
This is called phenomenal unity of consciousness.
implications for considerations in philosophy of
However, when the features belong to a single object,
mind and philosophy of perception, in particular on
they are experienced not only as copresent but as
such topics as theories of the unity of consciousness,
constituting one and the same object. This is known
cross-modal experiences, sensory integration, and
as object unity. We often switch between these kinds
the binding problem (Hardcastle 1998; Bayne 2010;
of unity as well as between their various degrees
Mroczko-Wąsowicz 2011; Macpherson 2007, 2011a;
of strength as a result of attention processes and
Mroczko-Wąsowicz and Werning 2012) as well as
mechanisms of multisensory integration (Deroy 2014;
debates on mental architecture, modularity of mind,
cf., Bayne 2010; Mroczko-Wąsowicz 2011). In synes-
and cognitive penetrability of perception (Fodor 1983;
thetic perception these unities take place together
Pylyshyn 1999; Baron-Cohen et al. 1993; Segal 1997;
as phenomenal object unity of consciousness. This
Macpherson 2012; Siegel 2012; Mroczko-Wąsowicz
means synesthetes are able to assign additional
and Nikolić 2014). A good theory must be able to
phenomenal properties to certain objects. The
accommodate how a system usually works and how
phenomenon of synesthesia exhibits a multitude of
it deviates from such functioning. Synesthesia can be
experiential levels driven by sensory, motor, bodily,
37
a rich diversity of phenomenal contents permanently
connectivity between distinct brain areas which
and perceptually bound together within a single uni-
facilitates mappings between concepts, experiences,
fied conscious experience (Mroczko-Wąsowicz 2011).
and behaviors (Ramachandran and Hubbard 2001).
These contents represent both high- and low-level
Synesthetic abilities are quite common among artists
properties. In this way, synesthesia transgresses the
who use creative multisensory interactions in their
boundaries between perception, action, and cogni-
works. There are various methods of communicating
tion, i.e., between capacities traditionally considered
synesthetic experiences. The preferred techniques
in philosophy, psychology, and cognitive science
deployed by synesthetic artists allow those aspects of
as independently operating domains (Mroczko-
their synesthesia that are otherwise inexpressible
Wąsowicz and Werning 2012; Mroczko-Wąsowicz
to be expressed (Steen 2001, 203). The artworks
2015). The resulting conscious experiences in
inspired by synesthesia may convey some information
synesthesia are typically synchronically unified,
about the phenomenology of this condition. As has
meaning that all the sub-experiences at a given time
already been mentioned, artistic practice and work
are present simultaneously, forming an overall
in aesthetics are relevant to debates in philosophy
phenomenal perspective — a single encompassing
of mind and philosophy of perception, especially
phenomenal state.
those concerning imagination, perception, judgment,
be co-conscious parts or aspects of the subsuming
and the construction of meaning (Noe ¨ 2015). Investigating these issues while taking synesthesia
state which has a conjoint phenomenology, or a joint
research into consideration seems likely to benefit
The different component experiences are said to
phenomenal content (Shoemaker 1996, 2003; Bayne
all the disciplines involved. This provides a sufficient
and Chalmers 2003; Tye 2003; Raymont and Brook
reason to study the phenomenon of synesthesia
2009). As such, they are intimately linked to and
from different epistemic perspectives using
integrated from the perspective of the self. The phenomenal object unity experienced at a particular point in time (e.g., when enjoying synes-
philosophy, science, and the arts. Such an interdisciplinary cooperation may well generate novel insights into human cognition.
thetic inducer-concurrent couplings) is a functional property, multi-realizable by a variety of synesthetic conditions. Also, it is not a binary feature of conscious experience, either fully present or not at all. The gradual instantiation of the unity becomes evident when comparing the experiences of two main groups of synesthetes (associators and projectors) and their different spatial frame of reference, including different perceptual salience as well as when contrasting synesthesia with synchronesthesia, i.e., common multisensory perception of non-synesthetes. This suggests that such a phenomenal coherence / qualitative congruence comes in degrees which may correspond to various degrees of cross-modal binding. Although the links between synesthesia and creativity remain unclear, there is some evidence of a tendency for synesthetes to be more engaged in art (Ward et al. 2008; van Campen 2007; Rothen and Meier 2010). In addition, some researchers suggest
THEORIES ON DIGITAL SYNESTHESIA
38
Aleksandra Mroczko-Wąsowicz
that synesthetes and artists inclusively exhibit greater
A Philosophical Perspective on Unified Conscious Experience in Synesthesia: Insights from Philosophy of Perception and Aesthetics
emotional, and conceptual systems. Thus, it combines
Acknowledgments This work was supported by the Ministry of Science and Technology (project: MOST104-2628-H-010002-MY3).
References Baron-Cohen, S., Wyke, M., and Binnie, C. (1987). Hearing words and seeing colours: an experimental investigation of a case of synaesthesia. Perception 16, 761–767. doi:10.1068/p160761 Baron-Cohen, S., Harrison, J., Goldstein, L. H., and Wyke, M. (1993). Coloured speech perception: Is synaesthesia what happens when modularity breaks down? Perception 22, 419–426. Bayne, T. (2009). Perception and the reach of phenomenal content. Philos.Q. 59, 385–404. doi:10.1111/j.1467-9213.2009.631.x Bayne, T. (2010). The Unity of Consciousness. Oxford: Oxford University Press. Bayne, T. and Chalmers, D. (2003) What is the unity of consciousness? In A. Cleeremans (Ed.) The Unity of Consciousness: Binding, Integration and Dissociation. Oxford: Oxford University Press. Brogaard, B. (2013). Serotonergic hyperactivity as a potential factor in developmental, acquired and drug-induced synesthesia. Frontiers in Human Neuroscience 7:657. doi: 10.3389/fnhum.2013.00657 Chiou, R., and Rich, A.N. (2014). The role of conceptual knowledge in understanding synesthesia: evaluating contemporary findings from a “hub-and-spokes” perspective. Frontiers in Psychology 5:105. doi:10.3389/fpsyg.2014.00105 Cohen Kadosh, R., Henik, A., Catena, A., Walsh, V., & Fuentes, L.J. (2009). Induced Cross-Modal Synaesthetic Experience Without Abnormal Neuronal Connections. Psychological Science, 20, 258–265. Deroy, O. (2014). The Unity Assumption and the Many Unities of
Consciousness. In D.J. Bennett and C.S Hill (eds.) Sensory Integration and the Unity of Consciousness. Cambridge, MA: MIT Press, 105–124.
Macpherson, F. (2011a). CrossModal Experiences. Proceedings of the Aristotelian Society 111 (3pt3):429–468.
Deroy, O. (2015). Can sounds be red? A new account of synaesthesia as enriched experience. In Coates, P. & Coleman, S. (eds.). Phenomenal qualities. Oxford: Oxford University Press.
Macpherson, F. (2011b) The admissible contents of experience. In: Macpherson, F. and Hawley, K. (eds.) The Admissible Contents of Experience. Wiley-Blackwell.
Deroy, O. and Spence, C. (2013). Training, hypnosis, and drugs: artificial synaesthesia, or artificial paradises?. Frontiers in Psychology 4:660. doi: 10.3389/fpsyg.2013.00660 Dixon, M. J., Smilek, D., Cudahy, C., and Merikle, P. M. (2000). Five plus two equals yellow. Nature 406, 365. Dixon, M.J., Smilek, D., Duffy, P. L., Zanna, P. M., and Merikle, P. M. (2006). The Role of Meaning in GraphemeColour Synaesthesia, Cortex 42, 243–252. Fodor, J. A. (1983). The Modularity of Mind. Cambridge, MA: MIT Press. Gray, R. (2011). On the nature of the senses. In F. Macpherson (ed.) The Senses: Classic and contemporary philosophical perspectives, (New York: OUP), 243–260. Hardcastle, V. (1998). The binding problem. In W. Bechtel and G. Graham (eds.) A Companion to Cognitive Science, Oxford: Blackwell Publishers. Keeley, B. L. (2013). What exactly is a sense? In J. Simner and E. Hubbard (eds.) The Oxford Handbook of Synaesthesia, (Oxford: OUP), 941–958. Luke, D.P. and Terhune, D.B. (2013). The induction of synaesthesia with chemical agents: a systematic review. Frontiers in Psychology 4:753. doi: 10.3389/fpsyg.2013.00753 Locke, J. (1689). An Essay Concerning Human Understanding, Clarendon Edition, ed P.H. Nidditch (Oxford: Clarendon Press); (repr. 1979). Macpherson, F. (2007). Synaesthesia, Functionalism and Phenomenology. In: De Caro, M., Ferretti, F. and Marraffa, M. (eds.) Cartographies of the Mind. Philsophy and Psychology in Intersection. Series: Studies in brain and mind, 4. Springer Verlag.
Macpherson, F. (2012). Cognitive penetration of colour experience: rethinking the issue in light of an indirect mechanism. Philos Phenomen Res 84, 24–62. Mann, H., Korzenko, J., Carriere, J.S.A., and Dixon, M.J. (2009).Timespace synaesthesia – a cognitive advantage? Consciousness and Cognition 18, 619–627.doi:10. 1016/j. concog.2009.06.005 Martino, G., and Marks, L.E. (2001). Synesthesia: strong and weak. Curr. Dir. Psychol. Sci. 10, 61–65 10.1111/1467-8721.00116 Matey, J. (2014). Can Blue Mean Four? In D.J. Bennett and C.S Hill (eds.) Sensory Integration and the Unity of Consciousness. Cambridge, MA: MIT Press, 151–170. Meier, B. (2013). Semantic representation of synaesthesia. Theoria et Historia Scientiarum 10, 125–134. http:// dx.doi.org/10.12775/ths-2013-0006 Mroczko-Wąsowicz, A. (2011 /2013). The Unity of Consciousness and the Phenomenon of Synesthesia [Die Einheit des Bewusstseins und das Phänomen der Synästhesie]. Published doctoral dissertation, Johannes Gutenberg University of Mainz.
Mroczko, A., Metzinger, T., Singer, W., and Nikolić, D. (2009). Immediate transfer of synesthesia to a novel inducer. Journal of Vision 9, 25, 1–8. doi: 10.1167/9.12.25 Mroczko-Wąsowicz, A., and Nikolić, D. (2013). Colored alphabets in bilingual synesthetes. In: J. Simner and E. Hubbard (Eds.) Oxford Handbook of Synesthesia, Oxford: Oxford University Press, 165–180. Mroczko-Wąsowicz, A., and Nikolić, D. (2014). Semantic mechanisms may be responsible for developing synesthesia. Frontiers in Human Neuroscience 8:509. doi: 10.3389/fnhum.2014.00509 Mroczko-Wąsowicz, A., and Werning, M. (2012). Synesthesia, sensory-motor contingency and semantic emulation: how swimming style-color synesthesia challenges the traditional view of synesthesia. Frontiers in Psychology 3:279. doi:10.3389/fpsyg.2012.00279 Nanay, B. (2015). Using Philosophy of Perception in Aesthetics. Aesthetic Investigations, 1 (1), 174–180. Nikolić, D., Jürgens, U.M., Rothen, N., Meier, B., and Mroczko A. (2011). Swimming-style synesthesia. Cortex 47, 874–879. Noe¨ , A. (2015). Strange Tools: Art. And Human Nature, New York: Farrar, Straus and Giroux / Hill and Wang. O’Callaghan, C. (1998). Seeing what you hear: Cross-modal illusions and perception. Philosophical Issues 18, 316–338. Pylyshyn, Z.W. (1999). Is vision continuous with cognition? The case for cognitive impenetrability of visual perception. Behav. Brain Sci. 22, 341–365. doi: 10. 1017/ s0140525x99002022
Mroczko-Wąsowicz, A. (2015). What Can Sensorimotor Enactivism Learn from Studies on Phenomenal Adaptation in Atypical Perceptual Conditions? – A Commentary on Rick Grush and Colleagues. In T. Metzinger & J. M. Windt (Eds). Open MIND: Philosophy and the Mind Sciences in the 21th Century. Cambridge, MA: MIT Press, 16(C), 1-17. doi: 10.15502/9783958570603
Ramachandran V.S., Hubbard E.M. (2001). Synaesthesia: A window into perception, thought and language. Journal of Consciousness Studies, 8 (12):3–34.
Mroczko-Wąsowicz, A. (forthcoming). Editorial: Perception-Cognition Interface & Cross-Modal Experiences. Insights into Unified Consciousness. Frontiers in Psychology/ Specialty: Consciousness Research.
Raymont, P., Brook, A. (2009). The Unity of Consciousness. In: B.P. McLaughlin, A. Beckermann, S. Walter (Eds.) The Oxford Handbook of Philosophy of Mind. Oxford: Clarendon Press.
39
Rothen, N., and Meier, B. (2010). Higher prevalence of synaesthesia in art students. Perception ,39(5), 718–720. doi:10.1068/p6680 Rothen, N., Nikolić, D., Jürgens, U.M., Mroczko-Wąsowicz, A., Cock, J., and Meier, B. (2013). Psychophysiological evidence for the genuineness of swimming-style colour synaesthesia. Conscious Cogn 22, 35–46. Sagiv, N., Dean, R.T., and Bailes, F. (2009). Algorithmic Synaesthesia. In: R.T. Dean (Ed.), The Oxford Handbook of Computer Music, New York: Oxford University Press, 294–311.
Segal, G. (1997). Synaesthesia: Implications for the modularity of mind. In S. Baron-Cohen and J.E. Harrison (ed.) Synaesthesia: Classic and Contemporary Readings (Blackwell, Oxford), 211–224. Shoemaker, S. (1996). Unity of consciousness and consciousness of unity. In The First-Person Perspective and Other Essays. Cambridge: Cambridge University Press Shoemaker, S. (2003). Conscious ness and Co-Consciousness. In: A. Cleeremans (Ed.) The Unity of Consciousness: Binding, Integration and Dissociation. Oxford: Oxford University Press, 59–71. Siegel, S. (2010). The Contents of Visual Experience. New York: Oxford University Press.
Siegel, S. (2012). Cognitive Penetrability and Perceptual Justification. Noûs, 46, 201–222. Simner, J. (2007). Beyond perception: synaesthesia as a psycholinguistic phenomenon. Trends Cogn. Sci. 11, 23–29. Steen, Carol (2001). Visions Shared: A Firsthand Look into Synesthesia and Art. Leonardo, 34 (3), 203-208. Tye, M. (2003). Consciousness and Persons: Unity and Identity. Cambridge, MA: MIT Press. van Campen, C. (2007). The Hidden Sense. Synesthesia in Art and Science. Cambridge: MIT Press. Ward, J., Huckstep, B., Tsakanikos, E. (2006). Sound-colour synaesthesia: To what extent does it use cross-modal mechanisms common to us all? Cortex, 42: 264–280.
Ward, J., and Sagiv, N. (2007). Synaesthesia for finger counting and dice patterns: A case of higher synaesthesia? Neurocase 13, 86–93. Ward, J., and Simner J. (2003). Lexical-gustatory synaesthesia: Linguistic and conceptual factors. Cognition, 89: 237–261. Ward, J., Thompson-Lake, D., Ely, R., and Kalinski, F. (2008). Synaesthesia, creativity and art: What is the link? British Journal of Psychology, 99, 127–14. Ward, J., Tsakanikos, E., and Bray, A. (2006). Synaesthesia for reading and playing musical notes. Neurocase 12, 27–34. doi:10.1080/13554790500 473672
A Philosophical Perspective on Unified Conscious Experience in Synesthesia: Insights from Philosophy of Perception and Aesthetics
Sagiv N., Ilbeigi A., Ben-Tal O. (2011). Reflections on synaesthesia,
perception and cognition. Intellectica 55, 81–94.
Aleksandra Mroczko-Wąsowicz
Rich, A.N., Bradshaw, J.L., and Mattingley, J. B. (2005). A systematic, large-scale study of synesthesia: Implications for the role of early experience in lexicalcolor associations. Cognition, 98, 53–84.
THEORIES ON DIGITAL SYNESTHESIA
40
Ideasthesia and Art
Danko Nikolić
Keywords ideasthesia, art, synesthesia, aesthetics, beauty, perception, colors, entertainment, science, balance
Abstract Ideasthesia can be defined as a phenomenon in which
underlie the creation and consumption of art. Here, an
activation of concepts produces phenomenal experi-
attempt is made to explain the psychology of art, as
ence. The present article is concerned with the
well as define art, based on the theory of ideasthesia.
relationship between ideasthesia and art. In the past,
According to the present theory, art happens when
it has proven difficult to come up with a comprehen-
the intensities of the meaning produced by a certain
sive definition of art. Equally difficult seems to be to
creation and the intensities of the experiences induced
understand which psychological processes specifically
by that creation, are balanced out.
41
The word ideasthesia comes from Ancient Greek
get the color of the Latin A. Much other evidence also
words idea (for concept) and aesthesis (for sensation).
exists (Simner et al. 2006, Novich et al. 2011, Chiou
Hence, the term ideasthesia means sensing concepts.
and Rich 2014, Sagiv et al. 2006, Ward and Sagiv 2007,
The theory of ideasthesia was initially developed on
Dixon et al. 2000, Nikolić et al. 2011). These results could not be explained by direct
people who have additional sensory-like experiences
connections between brain areas. Ideasthesia needed
to a stimulus that otherwise would not induce such
to be introduced. The theory of ideasthesia states that
experiences. For example, every letter of the alphabet
concepts precede sensory-like experiences in synes-
may induce the experience of a certain color (letter A
thesia (Nikolić 2009). Thus, it is first the concept of the
often happens to be red). And there are many other
letter A that is activated and then, as a consequence
forms of synesthesia (Cytowic and Eagleman 2009).
of the activation of that concept, a synesthetic color
Initially, it was believed that synesthetic experienc-
“lights up” in the minds of synesthetes. The theory
es occur as a result of direct connections between
of ideasthesia emphasizes a tight relationship between
corresponding parts of the brain. For example, the
concepts; i.e., our understanding of the stimuli and
brain area responsible for grapheme detection would
the experiences i.e., the way we feel about that
somehow (aberrantly) be connected to the brain area
world (Mroczko-Wasowicz and Nikolić 2014; van
responsible for color perception. However, this hy-
Leuween et al. 2015).
pothesis had to be abandoned over time as more and
Importantly, a concept is something fundamental-
more evidence suggested that synesthesia was not
ly different than a mere direct connection between
based on direct associations, but instead that con-
brain areas. Concepts are related to our capability of
cepts mediate those associations.
engaging our full intellect to understanding the world
For example, experiments showed that synes-
around us, and of acquiring novel insights about that
thetes would associate different synesthetic colors to
world (e.g., Fodor 1998). There is no consensus on
one and the same stimulus, depending on how they
how the brain implements concepts. One recent theo-
interpreted that stimulus (Dixon et al. 2006). It has also
ry proposes that we apply concepts whenever nerve
been shown that new synesthetic associations could
cells adapt to external stimuli; and thus, that we learn
be created within minutes — simply by giving new
new concepts as our neurons learn how to perform
meaning to a certain stimulus (Mroczko et al. 2009).
such adaptations. This theory, named practopoiesis,
For example, when synesthetes were exposed to an
presumes that the mind results from a hierarchy of
unfamiliar writing system they would associate colors
adaptive mechanisms (Nikolić 2015). According to
to new letters as soon as they learned which of the
practopoiesis, the “idea” component of ideasthesia
Latin letters they corresponded to; a new “A” would
involves fast processes of neural adaptation.
Danko Nikolić
the basis of research in synesthesia. Synesthetes are
Since its introduction, the theory of ideasthesia has
consistent associations. They discovered that subjects
been extended to everyday perceptual processes.
overwhelmingly agreed about many other properties
Milan et al. (2014) investigated what kind of personality
assigned to the shapes. For example, Kiki was nervous
traits people tend to assign to two different shapes,
and high-class. Bouba was lazy and easy going. They
known as Kiki and Bouba (Fig. 1; Köhler, 1947).
concluded that sensations produced by a “spiky shape”,
Before their research, it was well known that people
a “nervous personality”, “laziness”, much like those of
consistently assign the name Kiki to the spiky star and
the sounds making the name “Kiki”, are all connected
Bouba to the round blob. However, what Milan and
in an associative network. These types of associations
colleagues found was a much more extensive form of
are also known as cross-modal associations (e.g.,
THEORIES ON DIGITAL SYNESTHESIA
42
Ideasthesia and Art
Ideasthesia in Everyday Perception
Spence 2011); much like the color blue is perceived as
between semantics and sensations in such a way that
cold and red as warm. Importantly, as
the theory of ideasthesia would apply. This was the
Milan and colleagues noted, this network of
first proposal for ideasthesia being used outside of the
cross-modal associations had stark similarities to
realm of synesthesia and being generalized to every-
the semantic associative network (e.g., “doctor” is
day perceptions. Since then, further proposals have
associated with “nurse”). But these were associations
been made on how ideasthesia may help us better
among sensations, not among concepts. Therefore,
understand consciousness (Mroczko-Wasowicz and
they concluded that there is a strong similarity
Nikolić 2014; van Leuween et al. 2015).
Ideasthesia and Art Perhaps the most surprising consequence of intro-
offer a certain intellectual value? For example, what
ducing the concept of ideasthesia has been in its
is aesthetically different about the way Dostoyevsky
relationship to art. I have been surprised to witness
depicts Russian society in the 18/19th century and how
many artists adopting the concept in describing or
historians do so?
even naming their pieces of work. I have found these
Many attempts have been made at defining art,
concepts being used in a wide range of art forms
though this endeavor has proved difficult (Painter
including: paintings, music, photography, interactive
2002). The challenge is to make a distinction precisely
art, acting, installations, and even perfumes. When
between where the art is and where it ends. A
talking to some of those artists, I learned that they
successful definition would minimize false inclusions
overwhelmingly felt that ideasthesia somehow
and false exclusions, i.e., the number of works that
described the very process by which they created art.
would be accepted as art but would not be covered by
This adoption of ideasthesia by artists also pro-
the definition and vice versa, those that the definition
vokes a question: Can we use ideasthesia to formulate
would cover but that would generally be rejected.
a theory of art? Over time, I have come to believe that
A successful definition would also account for the
the answer to this question is affirmative. The relation-
subjectivity of art.
ship between the two opposing forces of ideasthesia,
The definition that follows also offers a theory
i.e., the concept and the sensation, can be used to for-
of art and is not made by an art historian, but by a
mulate a hypothesis about psychological events that
cognitive scientist. It may well be that the art prob-
underlie the process of either creating an art piece
lems presented here are incomplete and somewhat
or appreciating (i.e., consuming) it. I call the theory:
naïve. In any case, an effort to bridge these distant
ideasthesia balance theory.
fields — cognitive science and art theory — has
The key problem here is to uniquely distinguish art from all other, somewhat related human activities.
been made. The present theory is a psychological theory; it
For example, we can ask the question: What precisely
postulates what is happening within the minds of
distinguishes one novel that is generally judged as a
people when they experience art. Here, we try to
valuable piece of art from another novel that clearly
understand what the psychological processes are that
does not get such appreciation? Or, what is the
lead a person to declare some act or artifact as a
fundamental aesthetic difference between a “soap”
piece of art. The theory should apply to both the
opera as compared to a Shakespeare play? Likewise,
actual act of creating art and the consumption of art.
one can ask: What is it that distinguishes insightful
At the heart of the present theory lies the relation-
intellectual works that are unanimously considered
ship between meaning and experience as the two
not to be art, such as e.g., a doctoral thesis, from
forces of ideasthesia. By meaning, what I refer to here
other works that are considered a piece of art but also
far exceeds the type of meaning that can be expressed
43
matter what the insight is about. It can be about
linguistics. Here, under “extraction of meaning” from
academic content such as a mathematical theorem,
a stimulus we assume all of the changes made by our
but also about social life or even our basic perceptions
nervous system to prepare us for interaction with
such as colors, sounds, movements, or space. Whatever
that stimulus (Nikolić 2015); irrespective of whether
there is that we can process, memorize and under-
these changes can be described by words or not. In
stand, this same thing can also be probed by stimuli
ideasthesia, meaning is much wider than what can
that are either more or less meaningful. Sensation, on the other hand, as the second part of ideasthesia, is related to phenomenal experiences,
is largely involved in the extraction of meaning (e.g.,
also known as qualia. Sensations are about the way
Kuipers et al. 2013). Similarly, the number of con-
things feel and “are like”. It is about the redness of a
cepts that we have is far larger than can be named
red color, and the sourness of a lime. Sensations make
or expressed in words. Therefore, the meanings that
up our inner mental life and “light it up” so that life
relate to language are only a small fraction of the total
does not happen “in the dark”.
meaning machinery that our minds are equipped with. Consequently, by meaning I refer here to the full
To formulate the ideasthesia balance theory it is necessary first to establish that both meanings and
depth of relationship an art piece has to a person’s
sensations can vary in magnitude. Stimuli may be
knowledge. Every individual has a certain amount
more or less meaningful, and may have more or
of life experience, defining that person’s knowledge
less intensive sensations. For example, a light touch
about the life and world. This knowledge can be
to someone’s skin may not be noticed, and hence
described as an associative network of concepts, or
a sensation might be considered weak. In contrast,
alternatively as a hierarchy of concepts. The contents
a more forceful touch may produce a stronger
of knowledge include factual knowledge, episodic
sensation. Likewise, the same light touch may occur
memories of events that occurred in a person’s life,
accidentally in a crowded street and be thus meaning-
habits, general principles of understanding how the
less, or might indicate an important sign of affection
world works, our skills, personal values, learned
and hence carry a lot of meaning.
emotional responses, fears, hopes, individual philosophy of life, and so on. For the most part
In our mental life, meaning and sensation interact, and their relationship can be rich. We have discussed
this knowledge is unconscious, and hence not easy
cases above in which meaning induces sensation. We
to describe with words.
have also seen similarities between the network of the
These components of our knowledge do not stand
qualities of sensations and the network of concepts
independently, but are related and connected. One
underlying the extraction of meaning. But there are
meaningful aspect of any stimulus, be it art or not, is
more relationships between the two. Firstly, extracted
that which taps into the vast knowledge that a person
meaning can alter the perceived strength of sensation;
comes with and uses it to interact with the world. A
everything else being equal, a meaningful stimulus
stimulus that is particularly strong on meaning is one
is likely to induce a stronger sensation than a less
that probes our existing knowledge at a deeper level;
meaningful one. Also, the intensity of a sensation can
a meaningful stimulus powerfully affects the semantic
affect the process of learning, and can thus determine
structure that we already come with. Such a stimulus
the efficiency with which we acquire new knowledge
not only creates new memories, but makes us re-
(including new concepts).
organize the existing ones. It makes us form new con-
Ideasthesia balance theory states a particular
nections – new insights. A truly meaningful stimulus
relationship between the depth of meaning and the
makes us see some aspect of the world more clearly;
intensity of sensation: A creation that we are likely to
better, in a new light, in a more comprehensive way.
judge as art is one in which the meaningfulness and
To be declared as meaningful, it does not really
THEORIES ON DIGITAL SYNESTHESIA
the strength of sensation are well-correlated. In an art
44
Danko Nikolić
be verbalized. Hence, the theory takes into account the findings of neuroscience that the motor cortex
Ideasthesia and Art
merely by language — i.e., that which is understood by
piece, the moments (the components) that carry the
extreme point we can put entertainment, with
most meaning are also those that induce the strongest
excessive sensations and little meaning, and at the
sensations. If there is an event in a story, a sequence
other extreme point we can put various forms of
of notes in a melody, or a form in a sculpture that
scientific and technical work that are often loaded
induces the strongest sensations, this same event,
with meaning but are lacking in sensations (Fig. 4).
musical sequence and form must also be the one
Importantly, art falls exactly in the middle of that
that carries the most meaning. These moments and
continuum.
components have to be the places of insight that are perfectly combined with simultaneously inducing e.g.,
Comparison of art to entertainment: Entertainment
the strongest emotions. Therefore, art needs to effec-
runs on emotions, with an emphasis on emotions of
tively combine an insight with feeling.
positive valence. Negative emotions typically present
It is not that all human activities produce such a
an interlude to a resolution by positive ones (a happy
correlated induction of sensations and conceptualiza-
ending). Fear is followed by security; tension followed
tions. In fact, a perfect balance is difficult to achieve
by relaxation, and so on. In entertainment, this emo-
and requires hard work. It is comparably easier to
tional rollercoaster usually does not carry significant
induce strong sensations that do not have a strong
meaning. The consumer does not learn much. There
meaning attached. Also, it is comparatively easier
is no significant novel relation to the real world. In
to express meaning in a way that is quite dull with
entertainment, it does not matter whether the events
respect to the sensations that it evokes.
that unfold are realistic to life, whether the decisions
The relationships between meaning and sensa-
of characters are illogical, of whether the incidents
tions with respect to the balance of ideasthesia are
building the plot are physically impossible. The con-
illustrated in Fig. 2 – 5. Fig. 2 shows the entire
sumer of entertainment is there for the sensational
space of possible relationships between meaning
ride, not for insights and logical consistency. In con-
and sensations. However, there is only a narrow band
trast to entertainment, the sensations of an art piece
of relationships that can be considered as well
are combined with insights. The story in a drama may
balanced. Note that the absolute value of intensity is
help us learn something about real life. It may help us
not considered important; it is rather the balance of
understand ourselves or our friends. Often, art helps
the respective intensities that matters. Every art piece
make sense of emotions. For example, both James
is comprised of multiple parts and each part will carry
Bond and The Godfather movies are fictions that por-
more or less meaning or induce more or less sensation.
tray plenty of murders, explosions, fistfights, revenges,
These intensities need to be aligned as much as pos-
and erotic moments. However, only The Godfather
sible such that their idea vs. aesthesia relations lie all
tells a story of something that is realistic and from
along the gray diagonal in Fig. 2.
which one can learn about the real world: the fight for
A one-dimensional continuum of ideasthesia balance can be defined along the diagonal orthogonal to that of maximized balance (Fig. 3). This representa-
power, corruption, family relations, real conflicts in which individuals find themselves, and so on. For these reasons, art does not need to rely much
tion can indicate the overall idea vs. aesthesia balance
on happy endings. Art can also induce unpleasant
in a certain creation. The continuum
emotions, but if these emotions make sense in the
ranges from an extreme dominance of sensations
great scheme of things, i.e., if the events are mean-
(very little meaning) to an extreme dominance of
ingful overall, the consumer can take the insights that
meaning (scarce sensations).
come with the “ugly” emotions as a valuable outcome.
Such a one-dimensional representation can be
How can one handle the misfortune of the main
used to sort out various types of human creations
protagonist in Kafka’s Metamorphosis? This can
with respect to the dominance of either sensations
be done only by finding meaning in the story – for
or meaning, or their balance. For example, at one
example, by understanding it as a depiction of similar
45
Ideasthesia and Art
Danko Nikolić
Fig. 1
Fig. 2
THEORIES ON DIGITAL SYNESTHESIA
46
Fig. 1 Which one is Kiki and which one is Bouba? People overwhelmingly agree that the spiky star is Kiki.
Fig. 2 The space of possible relations between meaning and sensations. The contents of an art piece need to be aligned along the gray diagonal. Fig. 3 Definition of a one-dimensional continuum of ideasthesia balance.
Fig. 4 Entertainment, art, and science occupy different positions on the continuum of ideasthesia balance.
Fig. 3
Fig. 4
47
unsuccessful struggles happening to people in real
or anything else. However, the biggest difference from
life who may be suffering from a different nasty
art is not primarily that science is less capable of
misfortune of life than turning into a giant insect; such
generating such sensations (it is indeed less capable,
as for example a terminal disease. The beauty in this
but this is not the main difference) but that science
otherwise “ugly” story is in the balance between
cannot offer as good a balance between idea and
emotions and meaning.
aesthesia as art does. Scientific work cannot be made
Because art is meaningful in addition to often
such that it resides in the gray diagonal of Fig. 2.
giving us a good time, it provides us with a certain
The story of a scientific result is not told through a
value that exceeds that of entertainment — a value
set of rich sensory inputs. Instead, abstract (i.e.,
with a longer-lasting effect. Art builds us up, elevates
“boring”) language has to be used. In science, the point
us, and creates new experiences integrated into the
of highest meaning is not the point of strongest
semantic structures. It also expands these semantic
sensations induced. It is not that an emotional
structures. The new knowledge it imparts is likely to
equivalent of someone getting into a life-threatening
be useful in our lives at some later time; it is likely to
situation occurs at the moment of deriving the key
apply to events that actually occur to us. All else being
logical conclusion of an argument, or at the moment
equal, the person who has consumed art is likely to
of finding out that a statistical test was significant.
carry more wisdom than the one who has limited his or her experiences to mere entertainment.
Ideasthesia balance theory also tells us why art cannot be created by a forced combination of entertainment and scholarly work. One cannot create art by having characters in a soap opera spelling out
and make insights is not only a characteristic of art.
scientific facts. This would be unbalanced, and
Science and related disciplines create new knowledge
hence the resulting creation would be positioned all
as well but in a much more explicit way. Additionally,
over the graph in Fig. 2, not in the gray diagonal.
knowledge accessed through science is often more
For the same reason, one cannot simply tell jokes
direct, presented in a succinct form and is more
during a scientific presentation to turn science into art. Robert M. Pirsig (1999) argues in his book Zen and
accumulation of scientific knowledge tend to grow
the art of motorcycle maintenance that quality results
disconnected from everyday life with their specialized
from a combination of two factors: rational and
terminology, abstract concepts, and “dry” data. As a
romantic. I postulate that he is arguing something
result, they tend to produce less intensive sensations
similar to the balance of ideasthesia. The romantic
than art works. And when they do create sensations,
component relates to being “in the moment” and
these are not well-balanced with the implications of
defies rational analysis. In contrast, the rational
the scientific work. A discovery of a new molecule that
component is about details and understanding the
cures a certain disease is certainly very meaningful,
inner mechanics. Pirsig concludes that quality comes
but the way scientists describe such a molecule is by
when the two aspects are combined. In relation to
no means different from the description of any other,
ideasthesia, the romantic component may correspond
much less meaningful, molecule — it is just a formula.
to aesthesis, while the rational analysis may be the
In fact, the sensory experience is broadly similar for
idea part of ideasthesia. Therefore, to paraphrase
any formula describing any kind of molecule — and
Pirsig, high quality may be achieved when both
these experiences are not very rich at all. It is not that scientists don’t care about inducing
sensations and concepts are strong; that is, when ideasthesia is maximized.
sensation. They gladly hang onto any sensation-based aesthetics that can be associated with their abstract scientific work, be it simple symmetry, a nice fractal structure of data, an artificially-colored visualization
THEORIES ON DIGITAL SYNESTHESIA
48
Danko Nikolić
objective. However, scholarly works serving the
Ideasthesia and Art
Comparison to science: The ability to extract meaning
Implications The present theory can be used to offer an explana-
our definition of what art is. He offered an idea that
tion for the properties of art.
was revolutionary at the time.
Abstract art. A naïve view would be that abstract art
topic - that would be science. Instead, he painted a
does not carry meaning, and thus that it cannot fulfill
“black square” and he let people deal with the fear and
the criterion for the balance of ideasthesia. Where is
the insight. It is also possible that he could not have
the meaning in a bunch of circles and triangles on a
known for sure how his experiment would work out
Malevich did not write an academic essay on this
painting? Or, where is the meaning in one sequence
in the end, and may have had to deal with the same
of musical notes? The fact is that our brain extracts
fears himself. This combination of courage and insight
meaning at a level that cannot be verbalized. Meaning
makes this abstract painting a valuable piece of art.
does not require the ability to consciously express the contents. A stimulus is meaningful if it makes our
Subjectivity of art. A judgment of whether something
brain work, if it attracts our attention, and if it forces
is or is not art is highly subjective. For any particular
us to process it in some way. An input is more mean-
piece, there is no full consensus among experts. A
ingful if it requires more effort in adjusting to it. A lack
strong subjective component exists when valuing art.
of meaning comes from stimuli that we ignore or do
This is something that is perfectly expected according
not notice. Also, when we actively suppress input e.g.,
to the ideasthesia balance theory. The theory tells
filtering out one disturbing conversation in an effort
us that the artistic value of a creation can only be
to hear another, we can speak of little or no meaning
judged by the ideasthesia balance evoked within
being extracted from the suppressed stimulus.
an individual. Both the sensational and the meaning
Hence, abstract art can induce much meaning
aspects depend on the knowledge of a given
processed at a level difficult to express verbally.
person, and each individual is different. Hence, each
Also, abstract art can induce sensations. When these
individual will respond somewhat differently to
are combined well, one can achieve a balance of
any stimulus. A given creation may induce a strong
ideasthesia.
ideasthesia balance in some people and in others
However, the most powerful abstract art may be
it may not. In ideasthesia balance, art is exclusively
that which induces meaning and sensations that
defined by how a person reacts to it. This definition
transcend the modality in which the art has been
itself does not allow for any other definition of art
created. When Malevich painted his “black square”
that would be objective, i.e., that would work without
he did not primarily produce sensory sensations by
having to consider how people respond to it.
colors. Uniform black paint probably produces as little of such sensations as possible. The true sensations
Timelessness of art. Art pieces do not seem to
that he produced with that painting had to do with the
lose their effect over time. Classic paintings, classic
social aspects. Every other artist at that time would
music, classic texts; all of them continue to hold value
possibly fear stating that a black square is his own
as time goes by. In contrast, a piece of entertainment
artistic expression. One could easily be ridiculed. In
can explode in popularity for a very short period of
addition, every consumer of art or gallery custodian
time, and then it dies — it is forgotten as quickly as it
would have to deal with similar fears. And those
became popular. Moreover, the pieces that become
sensations were strong. At the same time, he offered
quickly popular end up not only less attractive soon
meaning; although not a verbal one. There was
afterward, but often even quite repulsive or annoying.
nothing much to be verbalized about a black square.
If a hit-driven radio station were to always play just
It was instead from his actions through which he
last year’s (last month’s) hits, it would quickly run
offered the idea that we can considerably broaden
out of listeners. In contrast, a classical radio station
49
can play music 100s of years old and still keep its
desirable as it enables exploration of the art piece —
audience happy. Who wants to watch a single episode
every fresh encounter leading to something new being
of a soap opera multiple times? And yet, owning a
discovered. In this way, an art piece educates.
copy of Beethoven’s Fifth and listening to it repeatedly
It is exactly this property of consistent balance
continues to evoke pleasure. How is that possible?
between sensation and meaning — i.e., balanced
How can ideasthesia balance explain the timelessness
ideasthesia — which makes us willing to preserve art
of art and the short-lived pleasures of entertainment?
pieces, to be surrounded by them, to protect them
We have to begin by noting that our brain changes with every single exposure to a stimulus. A stimulus
from being lost, and to encourage others to become exposed to them. Also those creations do not come
is always perceived and processed a bit differently
easily. They require hard work. But once it is done
the second time around. And the changes are due
right, they provide an incalculable value for many of us.
to learning about the stimulus. Our brain learns to predict what will come next, and habituates to familiar inputs (Johnston et al. 1990, Johnston and Schwarting 1997). As a result, there is less and less need to process the stimulus with every new exposure. The novelty is reduced. The intensity of sensations is reduced. The reason that art does not annoy us with repetition and entertainment does (or art annoys us much less) lies again in the ideasthesia balance. We have to ask: How would a given creation be represented in Fig. 2? An entertaining piece is, on the whole, located in the green area and therefore it is not well balanced. However, what makes entertainment nevertheless temporarily attractive is that, when Danko Nikolić
broken down into its components, it will have at least one component that touches the gray area - something like the red line shown in Fig. 5. It is likely that the section of a piece entering the gray zone of balance first will be the most salient one — and this may make the piece an instant hit. This most salient part will be the one that will wear off most quickly after repetition, however. Then the saliency will move on to other parts, and these other parts lie outside the gray region and hence lack beauty. Suddenly, the piece turns “ugly” and unpleasant.
Ideasthesia and Art
In contrast, an art piece is much more balanced across all of its components (something like the yellow line in Fig. 5). For that reason art continues to provide pleasure even after learning. As one component sinks into the background of habituation another component surfaces, but also offers balance, and hence value and beauty. Repeated exposure is then
THEORIES ON DIGITAL SYNESTHESIA
50
Fig. 5
Fig. 5 Sensation vs. meaning balance shown for hypothetical pieces of entertainment (red), art (yellow) and science (white), as they are broken down into their components. Only an art piece is consistently located in the gray area.
51
References Chiou R, Rich AN (2014) The role of conceptual knowledge in understanding synaesthesia: Evaluating contemporary findings from a “hub-and-spokes” perspective. Fron Psychol 5:105. Cytowic RE, Eagleman D. (2009) Wednesday is indigo blue: Discovering the brain of synesthesia. MIT Press. Dixon MJ, Smilek D, Cudahy C, Merikle PM. (2000) Five plus two equals yellow. Nature. Jul 27; 406(6794):365–. Dixon MJ, Smilek D, Merikle PM (2004) Not all synaesthetes are created equal: Projector versus associator synaesthetes. Cogn Affect Behav Neurosci 4:335–343.
DeWitt MJ. (1990) Attention capture by novel stimuli. Journal of Experimental Psychology: General. Dec;119(4):397. Johnston WA, Schwarting IS. (1997) Novel popout: An enigma for conventional theories of attention. Journal of Experimental Psychology: Human Perception and Performance. Jun;23(3):622.
Nikolić D (2009) Is synaesthesia actually ideaestesia? An inquiry into the nature of the phenomenon. In: Proceedings of the Third International Congress on Synaesthesia, Science & Art, pp 26–29.
Simner J, Mulvenna C, Sagiv N, Tsakanikos E, Witherby SA, Fraser C, Scott K, Ward J (2006) Synaesthesia: the prevalence of atypical cross-modal experiences. Perception 35:1024–1033.
Nikolić D, Jürgens UM, Rothen N, Meier B, Mroczko A (2011) Swimming-style synesthesia. Cortex 47:874–879.
Spence C (2011) Crossmodal correspondences: A tutorial review. Atten Percept Psychophys 73:971–995.
Köhler W (1947) Gestalt psychology: An introduction to new concepts in modern psychology. New York: Liveright.
Nikolić D. (2015) Practopoiesis: Or how life fosters a mind. Journal of theoretical biology. May 21; 373:40–61.
Kuipers JR, van Koningsbruggen M, Thierry G (2013) Semantic priming in the motor cortex: evidence from combined repetitive transcranial magnetic stimulation and eventrelated potential. Neuroreport, 24(12), 646–651.
Novich S, Cheng S, Eagleman DM (2011) Is synaesthesia one condition or many? A large-scale analysis reveals subgroups. Journal of Neuropsychology 5:353–371.
Mroczko-Wąsowicz A, Nikolić D (2014) Semantic mechanisms may be responsible for developing synesthesia. Front Hum Neurosci 8:509.
Fodor JA. (1998) Concepts: Where cognitive science went wrong. Clarendon Press. Johnston WA, Hawley KJ, Plewe SH, Elliott JM,
Mroczko A, Metzinger T, Singer W, Nikolić D (2009) Immediate transfer of synesthesia to a novel inducer. J Vision 9.
Ward J, Sagiv N (2007) Synaesthesia for finger counting and dice patterns: a case of higher synaesthesia? Neurocase 13:86–93.
Painter C. (2002) Contemporary art and the home. Berg Publishers; Oct 1. Pirsig RM. (1999) Zen and the art of motorcycle maintenance: An inquiry into values. Random House. Sagiv N, Heer J, Robertson L (2006) Does binding of synesthetic color to the evoking grapheme require attention? Cortex 42:232–242.
Ideasthesia and Art
Danko Nikolić
Dixon MJ, Smilek D, Duffy PL, Zanna MP, Merikle PM (2006) The role of meaning in grapheme-colour synaesthesia. Cortex 42:243–252.
van Leeuwen TM, Singer W, Nikolić D. (2015) The merit of synesthesia for consciousness research. Frontiers in psychology 6.
THEORIES ON DIGITAL SYNESTHESIA
52
Transformation, Transliteration and Translation. Synesthesia and Multisensory Perception in Contemporary Visual Art Regine Rapp
Keywords transformation, transliteration, translation, transmodality, perception, process, algorithm, grapheme, code, monitor
The term “synesthesia”, coming from the Greek
rich 2013). Sina Trautmann-Lengsfeld claims that “the
“aisthesis” (“sensation”, “sensory impression”) and “syn”
stimulation of one sensory modality leads to [another]
(“together”), means the experience of two or more
additional, atypical sensory experience in the same or
sensory impressions at the same time. Synesthesia was
different modality” (Trautmann-Lengsfeld 2013).
not only prominent in the melding of the arts from
In my paper I want to focus on three remarkable
the Renaissance, Romanticism, Symbolism and 20th
artistic positions that approach digital synesthetic
century avant-garde; it is also a highly valuable param-
forms of transmodality in unique ways. I will intro-
eter in the field of contemporary art. Neuroscientist
duce how selected modes of intersensory perception
Hinderk Emrich stresses that for the investigation of
are stressed and visualized in artistic works. They are
synesthesia, the “phenomenon of transmodality” is
expressed either as transformation (sound and vision
important. Synesthesia means “creating processes” –
into space), as transliteration (text into imagination) or
and this is “never mono, always trans, inter, syn” (Em-
as translation (language into color codes).
53
so-called “form constants” (Heinrich Klüver). Klüver
sculpture with digitally projected moving images
did tests in 1920 on the effect of mescaline (peyote)
to create performing environments. Her work explores
consumption. Within his research of users’ individual
the interactions between physical space and moving
hallucinogenic reactions (very often seeing bright
images, and often uses technology and digital media
flickering lights), he called those geometric
as well as taking inspiration from immersive theater
patterns “form constants” and defined four groups:
and experimental film. Boyd’s work focuses on per-
(I) tunnels and funnels, (II) spirals, (III) lattices,
ception and the brain. Her recent complex project
including honeycombs and triangles, and (IV) cob-
The Point of Perception (Fig. 1) from 2009/13 explores
webs (Bressloff et al. 2002). Current researchers have
how much information the human brain needs in
gone further and hypothesized that these patterns
order to know what it is looking at. It was produced
could be a visualization of the brain’s structure: “In
in collaboration with neuroscientists Dr. Mark Lythgoe
general the images do not move with the eyes. We
and Dr. Beau Lotto from University College London,
interpret this to mean that they are generated in
and is both an art experience and a scientific arena
the brain. Here, we summarize a theory of their origin
on vision.
in visual cortex (area V1), based on the assumption
The installation uses moving images and built
that the form of the retino–cortical map and
environments to confuse depth perception and is
the architecture of V1 determine their geometry”
designed to act specifically on the human eye and
(Bressloff et al. 2002). So “form constants” can mirror
brain to create a space of uncertainty. Madi Boyd
the brain’s inner architecture during times of psycho-
describes her artistic work as combining “constructed
logical pressure – not only drug consumption, but
environments and projected films to create immersive
also mental stress such as sensory deprivation. For
installations investigating the interaction between
Boyd it was an important part of her artistic process
moving image, space, and the brain.” She sees “the
as it helped her to understand how the elements she
screen as sculpture and light as paint on the canvas
worked with (motion, light, pattern, form, space) were
of dark space” (Boyd 2013). A newly developed version of the project was presented at Art Laboratory Berlin in 2013 with the addition of sound and color, bringing another parameter of transmodality into the art piece. The
received and interpreted by the brain, allowing Boyd to better predict how someone might perceive and experience her work. The process of spatial transformation within this installation is remarkable: The use of the endless
idea was to assign musical notes and colors to
loop gives it a never-ending, almost hallucinogenic
specific points in the gridded space. The sound was
effect. Madi Boyd’s piece is a perfect example of the
created by composer Nye Parry, and was composed in
so-called filled-space installations: Enchantments,
a way described by Madi Boyd, who herself perceives
where there is “an overall environment with little or
a form of color-sound synesthesia. Boyd challenges
no escape route, the enchantment draws heavily
the viewer’s spatial perception with a complex video
on their theatrical roots, the suspension of disbelief
projection of geometric structures in a dark room.
being chief among these” (Rosentahl 2003). Taking
The slow movements of the work’s projected abstract
into consideration the theory of the “form constants”
forms not only match the ambient sound that refers
as the visualized architecture of the brain and the sus-
to the artist’s personal synesthetic perception. Boyd
pension of disbelief – one might argue that with this
also manages to visualize geometric patterns run-
installation piece we can experience a walk into our
ning across the installation’s dark walls and ceilings:
own, or even the artist’s, brain.
THEORIES ON DIGITAL SYNESTHESIA
54
Transformation, Transliteration and Translation. Synesthesia and Multisensory Perception in Contemporary Visual Art
Madi Boyd is an installation artist, fusing large-scale
Regine Rapp
1. Madi Boyd – Synesthesia as Spatial Transformation
2. Ditte Lyngkær Pedersen – Digital Synesthesia as Transliteration Ditte Lyngkær Pedersen is a multidisciplinary artist
actually.” There is also protagonist Emily’s statement,
working in video and installation, and whose work
where we perceive the particular power the sound of
often includes collaborators. She has invested in
words can have on synesthetes (quite physically and
deconstructing the intersections of subjective and
literally): “PURGATORY, when I hear the word, I see
objective perspectives, in order to pose alternative
a brown-green pile of feathery shapes, sticky to touch,
narratives and forms of representation. Her work
with leaping flames of liquid red. It tastes like a sweet
critically explores the dynamics of language,
cheese on bitter greens wrapped in an apple with
synesthesia, and identity. Her long-term art project
lemon squeezed on; shaped like an abstract figure
on synesthesia Why is Green a Red Word? began
of metal; life-size, that I feel by sticking my arms into
in 2003 and triggered her choice of incorporating
it and running my fingers along its unpredictable
explorations on synesthesia into her art practice.
turns. […]” (Lyngkær Pedersen 2013).
Since 2003 Pedersen has created an extensive
Pedersen chose a fascinating method of transliter-
video archive of interviews about the multi-sensory
ation between word and image (as in the French word
perception of synesthesia that documents the expe-
“image”): Although no speaker is visually depicted in
riences of individuals and at the same time makes the
the video, it is nevertheless her voice that talks about
unbridgeable gap between this topic and the audience
the individual association. The recipient is mesmerized
clear. Her project Why Is Green a Red Word? is com-
by the distinct spoken, but invisible, explanations – on
prised of interviews with synesthetes and scientists.
the screen there is nothing but the static, motion- and
Pedersen’s long term project includes several
colorless letters of the word “purgatory”. Pedersen
outstanding video pieces – for example, the concep-
convincingly achieves the moment of “creating pro-
tual video work What the Hell does Purgatory Look
cesses” (Emrich 2013). Her chosen mode of transliter-
Like? (Fig. 2). Although the monitor shows throughout
ation is full of “trans”, “inter”, and “syn” as the listener
its entire length the word “purgatory” in large black
can’t help but follow the rich lively explanations of the
letters on a white background, the recipient (viewer)
protagonists. This trans-literation, remarkably enough,
receives the main impulse via headphones: several
leads us away from semantic meaning.
voices talk one after the other about their individual synesthetic perception of the word purgatory. Most of the time, however, they refer to the visual characterization; the semantics are often secondary or even undermined. Selected answers by the protagonists show us a plasticity, and an asemantic, sound-oriented quality. Protagonist Karen states: “PURGATORY, resembles a prune, but the word is partly greasy like Vaseline Petroleum Jelly. The syllables account for this. The word ‘group‘ is also like Vaseline. You can spell ‘group‘ from the letters that make up Purgatory.” Another protagonist, Anne Constance Cooper, notes: “PURGATORY, is the colors of a day old bruise; purple and blue and black. The color has a slightly wavering, heavy sound, like the big 4th note of Beethoven’s 5th. Interestingly, it is a less pleasant word than hell,
55
3. Eva-Maria Bolz – Digital Synesthesia as Translation Eva-Maria Bolz is an artist, graphic designer and
synesthetic coloring for letters, numbers and whole
grapheme synesthete based in Berlin. In the center
words. With the help of an algorithm, selected texts
of her creative activity is the intensive study of
have been converted into large format color plates.
typography and its implementation. Bolz has based
These plates consist of thousands of color bars and
her abstract color system directly to her own graph-
include all the information of the text. For Bolz “the
eme synesthetic perception: For many years she has
texts no longer speak about the form of the grapheme;
collected and recorded her own relationship between
instead the content is transmitted exclusively through
text and color in the form of a digital diary. She has
the associated synesthetic colors“ (Boltz 2013). Here the artist undermines the stressed semantics and system, as in the color panels we lose the sense
computer to compare slight changes in perception
of the content but consider the piece as an “all-over”
Her work Der innere Monitor (The Inner Monitor)
abstract. To speak in semiotic terms: Here the signifier becomes the signified and vice versa – the content
is dedicated to an exploration of the relationship
overlaps with the form. There is an interesting dualistic
between color, text and visual perception. Due to
aesthetic in Bolz’ aesthetic translation project: The
her grapheme synesthesia, she feels an unchanging
work Der innere Monitor appears as minimal due to
association of colors to numbers, letters, as well as
its strict systematic outline, but also as highly
selected whole words. Perception becomes a filter
playfully poetic, due to her choice of literary text. The
through which letters, words – text in itself – are
strict rules of translating letters into color (grapheme
translated into colors and transformed from a set of
synesthesia) oppose the humorous tone of the
well-known characters into a message that can be
work, echoing the wit in Ionesco’s play with nonsense
detected by means of a particular synesthetic
and grotesquery. We finally feel challenged by the
sensibility. Her project Der Innere Monitor follows her
opposition of dry linguistic phonetic sentences on
perception that colors and letters form a specific
the one hand, and the intimacy of the artist’s own
code through which a text can be translated into
inner color system on the other: by means of a
blocks of color. Each letter corresponds to a specific
code, the artist’s individual visual system has been
color. The artist deliberately uses texts that contain
translated into plates of color.
intense color descriptions, such as Oscar Wilde’s The Nightingale and The Rose. In the exhibition in spring 2013 at Art Laboratory Berlin, Bolz presented five selected texts in the form of large color plates with texts by Oscar Wilde, E. T. A Hofmann, Joris-Karl Huysmans, Eugène Ionesco and Marie-Luise Kaschnitz. For Bolz, the work “is based on the hypothesis that grapheme, and in particular the lexical synesthesia, enables the visualization of text content for a fixed system of meaning. This is also based on individual experiences, but is constant and involuntary, so is not an expression of current metaphorical allocation“ (Boltz 2013). The specific method is noteworthy: over several years she has systematically recorded her
THEORIES ON DIGITAL SYNESTHESIA
56
Regine Rapp
and generate a continuity of color-text relationship.
Transformation, Transliteration and Translation. Synesthesia and Multisensory Perception in Contemporary Visual Art
regularly mixed the particular color tones of letters, words, days of the weeks or months over time at her
A
A Ditte Lyngkær Pedersen, What The Hell Does Purgatory Look Like?, 2005, Video, 8 Min., exhibition view, Art Laboratory Berlin, 2013 | Installation view, Art Laboratory Berlin, 2013 Photo: Tim Deussen, 2013
B Madi Boyd, The Point of Perception, 2009/13, installation view, Art Laboratory Berlin, 2013 | Installation view, Art Laboratory Berlin, 2013 Photo: Amin Akhtar, 2013
B
57
Boyd, Madi. 2013. “The Point of Perception. On Collaborations Between Artists and Neuroscientists.” Presentation at Art Laboratory Berlin’s conference Synaesthesia.
Discussing a Phenomenon in the Arts, Humanities and (Neuro-)Science, Berlin, July 2013. Accessed January 30, 2016. http://artlaboratory-berlin. org/html/de-event-17.htm Bolz, Eva-Maria. 2013. In conversation with Regine Rapp. See also
exhibition text - Accessed January 30, 2016: http://artlaboratory-berlin. org/assets/pdf/Syn%204_Pressetext_2013_5_15.pdf
(Neuro-)Science, Berlin, July 2013. Accessed January 30, 2016. http://artlaboratory-berlin.org/html/ de-event-17.htm
Bressloff, Paul C. et al. 2002. “What Geometric Visual Hallucinations Tell Us About the Visual Cortex.” Neural Computation, The MIT Press, 14 (3): 473–491.
Lyngkær Pedersen, Ditte. 2013. Why is Green a Red Word? Artist Book. Published by Art Laboratory Berlin
Emrich, Hinderk. 2013. “Synaesthesia, Synaisthesis and the Enhancement of Coherence.” Keynote speech at Synaesthesia. Discussing a Phenomenon in the Arts, Humanities and
THEORIES ON DIGITAL SYNESTHESIA
Rosentahl, Mark. 2003. Understanding installation Art. From Duchamp to Holzer. New York München: Prestel. 33. Trautmann-Lengsfeld, Sina A. 2013. “Multisensory Processing in
58
Synaesthesia.” Presentation at Art Laboratory Berlin’s conference Synaesthesia.
Discussing a Phenomenon in the Arts, Humanities and (Neuro-)Science, Berlin, July 2013. Accessed January 30, 2016. http://artlaboratory-berlin.org/html/ de-event-17.htm
Transformation, Transliteration and Translation. Synesthesia and Multisensory Perception in Contemporary Visual Art
References
Regine Rapp
C Eva-Maria Bolz, Der innere Monitor / Oscar Wilde, 2013, Digital printinstallation view, Art Laboratory Berlin, 2013
Sensing Digital Art: Aesthetic Acts Beyond (the two) Synesthesia(s)
Chris Salter
Keywords touch, digital art, immersive art, multisensory integration, sensory substitution, stimuli, sensation, perception
It seems that everywhere we look, hear and feel in
sense-based information” and that haptics in
contemporary art and life, digital computational
particular would enable us to “feel the surface of
technologies are quickly overtaking our senses;
produce”, or eventually “virtually” high-five a
particularly our sense of touch. While Apple laptops,
hologram of Tupac Shakur (Schwartz 2012). More
for example, have long been adorned with seeing
recently, a NY Times feature on “The Future of Touch”
through their “retinal” displays , they have more
argued that “to interact with the world in any mean-
recently began to employ “Force Touch” and “Taptic
ingful way, we have to use the sense of touch” and
engines,” the latter of which aims to transform the
that haptic technologies would be the breakthrough
body itself into an interface.
in letting “people feel things that are not actually
In 2012, IBM’s “cognitive computing” research program’s “5 in 5” argued that in five years, machines could “extend our ability to gather and process
there” (Fergusson and Naudziunas 2015). I am zeroing in on touch here not simply because it appears to be the latest sense modality to become
59
the subject of (and subject to) technological buzz and
view, mainly focusing on two sense modalities in
hype. Touch, after all, has long been viewed as a
relationship to another . In Spence, Calvert and Stein’s
forgotten sense; one that as cultural historian
monumental volume on multisensory processes
Constance Classen claims “often remains unspoken
(2004), for example, the vast majority of articles ex-
and, even more so, un-historicized” (Classen xi, 2012).
amine two modalities crossing over each other. The
Increasingly integrated into phones, laptops, game
fabled McGurk effect, in which vision alters speech
controllers, theater and car seats, robotic arms
perception when the auditory component of one
and elsewhere, it seems that the science of haptics,
sound is paired with the visual component of another
applying force to the skin to deliver feedback and
sound, leading to the perception of a third (hearing ba
information, is now on the verge of becoming a new
and seeing ga results in perceiving da), involves vision and audition. Likewise, the pioneering cross-modal
Yet, touch is also interesting because it is one of
research from neuroscientist Shinsuke Shimojo also
those strange sensory modalities that appear to “cross
focused on only two modalities, namely, how sound
over” or “interfere” with the other senses. As a field
can alter the temporal and spatial structure of vision
of vibration, touch can suddenly be perceived as
(Spence, Calvert and Stein 2004, 27– 34). The notion of sensory substitution, made famous by Mexican neuroscientist Paul Bach y Rita’s work
audible range. Even Aristotle (2010), who is blamed
on brain plasticity in the 1960s (Bach-y-Rita, 1969)
for the classic reduction of sense perception into
likewise focused mainly on two sense modalities. In
five discrete categories, acknowledged that the neat
sensory substitution, one human sense is used to
definitions of organ to phenomena (correspondences
receive information normally associated with another
between sense organs and things sensed) was not
sense, for example, Bach-y-Rita’s TVS (Tactile Vision
so simple, arguing that “if touch is not one sense but
Substitution) in which a matrix of electro-tactile stim-
many,” then the fact that the “tangible sensibles are
ulators works to convey spatial information normally
also many … involves perplexity.” Thus, although it
associated with vision to the skin. Now a commercial
seems to occupy the surface of the skin, touch for
industry of assistive devices, sensory substitution
Aristotle may also be internal, involving many
usually implements the substitution of the visual sense
opposing (yet binary) sensations such as hot/cold,
in a machine through either auditory or tactile senses,
wet/dry, hard/soft and so on (2010, 69).
but rarely using more than two modalities.
In the social and natural sciences, however, it seems that any strange aberration of sense perception,
Even our current obsession with the almost mythical notion of synesthesia seemingly follows
like Aristotle’s understanding of touch, is perplexing
this dualistic pattern. The Oxford Handbook of
since it does not neatly fall into one modality and
Synesthesia defines synesthesia as “a neurological
always follows dualistic, dyadic patterns of vision/
condition which gives rise to extraordinary sensations”
hearing, touch/hearing and others. Many studies of
(Simner and Hubbard 2013, xxi), in which the “stim-
so-called multisensory integration, defined by Stein
ulation of one sense modality automatically evokes
and Meredith as a “statistically significant difference
a perception in an unstimulated modality” (2013, 3).
between the number of impulses that are evoked by
Even though synesthesia describes the “merging of
a cross-modal combination of stimuli and the
the senses,” a majority of researchers see this merg-
number of impulses evoked by the most effective
ing in pairs where an “inducer” in one sense triggers a
of these stimuli individually” (Calvert, Spence and Stein
consistent concurrent sensation in another modality:
2004, 248) purportedly claim to critique the “sense
color to grapheme (the most common form of
by sense” approach of traditional perceptual
synesthesia), mirror touch or lexical-gustatory (the
psychology and neuroscience. But most of this
rarest in which associations are formed between
research still emphasizes a dualistic sensory world-
words and tastes), among dozens of others.
60
Chris Salter
acoustic when it enters into the limited range of human hearing, moving from below 20 Hz into the
Sensing Digital Art: Aesthetic Acts Beyond (the two) Synesthesia(s)
socio-cultural paradigm.
But as anthropologist of the senses David Howes
synesthesia through what is still mainly a history of
reminds us (2013), synesthesia also operates within
audio-visual correspondences, from color organs to
another duality. Neuroscience, which is enjoying a
the Whitney-inspired slit-screen effects set to Ligeti’s
kind of pre-eminent status among the sciences at
Atmospheres and Requiem in the penultimate
the moment, emphatically distinguishes between
psychedelic conclusion of Kubrick’s 2001.
two forms of synesthesia: so-called “strong” synesthesia, defined as the brain-based, mainly congenital
In fact, as we dig deeper, artistic events that use the senses as material for the production of “blocs of
or “genuine, developmental” type in which subjects
percepts, sensations and affects” (Deleuze and
report synesthesia for as long as they remember, and
Guattari 1994, 167) more often than not transcend
“weak” synesthesia, which describes “milder forms of
such sensory dualisms. They introduce “perplexity”
cross-sensory connections revealed through language
into what Howes identifies as worlds “beyond and
and perception” (Martino, et al. 2001, 61).
beneath the five senses.” Beyond the five senses
For Howes, such weak synesthesia is not weak
invokes aesthetic experimentation into phenomena
at all, but rather constitutes the notion not only as
like the supernatural and the paranormal, while
a neurological practice but also as a cultural one.
“beneath the five senses” indicates the worlds of
Indeed, cultural practices that involve cross-modality,
“the visceral and the molecular” (Howes 2009, 22).
such as the ritual use of hallucinogens by the Desana
Indeed, as the 20th century theater sorcerer Antonin
and Tucano Indians of the Northern Colombian
Artaud wrote, a theater of cruelty would be one that
Amazon region, already articulate a merge between
is “addressed to the entire organism,” in which tech-
the biological, social-technical and cultural
nologies like light, sound and other scenographic
(Howes 2013, 156 – 159).
elements would produce “sensations of heat, cold,
Given that this volume focuses on current artistic practices that explore the intersections among synes-
anger, fear, etc.” (Artaud 1958, 95). The binary, switch-like nature of the digital also
thesia, digital art and aesthetics, it would seem that a
participates in the propagation of such sensorial
discussion of why the notion of synesthesia has played
dualism, giving short shrift to the complex entangle-
such a historically significant role in art practice would
ments that take place in the interdependent processes
also be in order. In fact, much has been written about
of stimuli, sensation and perception. That artists
the cultural synesthetic practices of artists as far-
increasingly employ digital technologies that repre-
ranging as Baudelaire, Scriabin, Kandinsky, Varèse,
sent stimulations and sensations in binary numbers
John Whitney or Gyorgy Ligeti, in which their interests
and then re-render them in strictly analog (that is,
in synesthetic processes catalyzed the development
analogous) form from those binary numbers in no way
of entirely new artistic styles and movements (van
implies that such processes can be reduced down to
Campen 2013, 631– 632). As is the case with touch,
simply numerical representations.
these histories have equally been silenced, with art
As Cretien van Campen argues, the immersive
historian Simon Shaw-Miller arguing that synesthesia
nature of contemporary art operates not just in
is denied by an art-historical frame in which “the eye is
sense categories, but on the body itself (van Campen
mute and the ear cannot see” (Shaw-Miller 2013,1).
2013, 635). The use of pressure, temperature,
Shaw-Miller is one of a handful of art historians
tension, proprioception, nociception and other felt
aiming to correct the academic tendency to divide
sensations by artists that combine, mix, intensify
the senses up among different discursive formations:
and diminish over time, generating unknown and
disciplines and faculties which split up art history from
uncertain transmutations, densities, colors, rhythms
musicology, “the sounds of art and the sight of music”
and “zones of intensities” (Varèse 1966, 13) goes
(2013, 1). Yet upon closer observation (or sensation),
far beyond the dualistic nature of correspondence
artistic practices do not usually hold to even the
between vision and audition, vision and touch,
supposedly current revisionist art-historical focus on
smell and taste.
61
In their 1991 work Divina Commedia: Praxis for
beyond the synesthetic evocation of “hearing shapes”
Death, for example, the Japanese artists/researchers
or “feeling vision,” and instead generate experiences
Masayuki Towata and Yasuaki Matsumoto designed
of a threshold nature in which sounds, light and
a completely bodily immersive installation in which
the thickness of space are felt moving around the
visitors would don specially designed suits and, while
skin and the shape of perceived time and space are
floating inside a 30 cm deep pool of edible gel (Iwaki
twisted inside out. Thus, despite its long (and still
2013, 217), experience a choreographed sequence of
ignored) historical trajectory, artistic work operating
flashing lights, generated from an array of computer-
at the borders between new technologies and new
controlled Dataflash strobes hung directly above.
morphologies of sensation might do well to bypass
Aiming to “loosen their sensory-motor connections,”
its supposedly radical synesthetic genealogies, in-
Towata and Matsumoto’s work was inspired by the
stead embracing Artaud’s challenge to create art as a
float tanks of John Lilly as well as research into
spectacle “immersed in a profound intoxication which
near-death experiences, a term coined by the
restores to us the very elements of ecstasy” (1958, 65).
American medical doctor Raymond Moody. Although this installation which explored the “grey zone between death and life” utilized both acoustic and visual media, we would be hard-pressed to describe Chris Salter
the almost out-of-body experiences and nearthreshold effects experienced by audience members (one of whom claimed to have “met Buddha” during the experience) as merely synesthetic (2013, 219). Later artistic works like artist Kurt Hentschläger’s ganzfeld-inspired installation Zee (2008) — in
Sensing Digital Art: Aesthetic Acts Beyond (the two) Synesthesia(s)
which audiences are plunged into blinding fields of stroboscopic light and impenetrable fog — or Sonia Cilari’s audience-performer intensive Sensitive to Pleasure (2010), in which the participant is caught up in near darkness, interacting with an indistinguishable body inside a room that, unbeknownst to them, results in the triggering of mild but persistent electrical pulses on the artists’ arms and chest outside of the space (http://nimk.nl/blog/sensitivetopleasure/) also go beyond the dualistic notions of synesthesia, instead generating full-body affects in which the corporeal sense of space and time are radically and inexplicably altered. Similarly, our performative environment Ilinx (Lamontagne, et al. 2015) where audience members wear wirelessly enabled, full-body haptic clothing inside an audio-visual environment which blurs together the modalities of sight, sound and touch, also results in a panoply of sensory impressions that generate simultaneous proprioceptive, haptic, acoustic, (semi)-visual, equilibrioceptive and temporal feelings and affects. These zones of intensities go
THEORIES ON DIGITAL SYNESTHESIA
62
A
B
A Sonia Cillari: Sensitive to Pleasure (2010–2011) Performative electric-field sensing and sound environment Photo: Sara Tirelli
B Kurt Hentschläger: ZEE (2008) Immersive audiovisual environment Photo: Kurt Hentschläger
63
Aristotle. 2010. De Anima. Trans. Mark Shiffman. New York: Focus Philosophical Library. Artaud, Antonin. The Theater and its Double. Trans. M.C. Richards. New York: Grove Press, 1958. Bach-Y-Rita, Paul, et al. 1969. “Vision substitution by tactile image projection.” Nature 221: 963 –964. Calvert, Gemma, Charles Spence and Barry Stein. 2004. Handbook of Multisensory Processes. Cambridge, MA: MIT Press. Cillari, Sonia. 2010. http:// www.soniacillari.net/Sensitive_ to_Pleasure.htm. (Accessed November 2, 2015)
Classen, Constance. 2012. The Deepest Sense: A cultural history of touch. Champagne-Urbana: University of Illinois Press. Deleuze, Gilles, and Félix Guattari. 1994. What is Philosophy? New York: Columbia University Press. Fergusson, Rebekah and Jessica Naudziunas. September 9, 2015. “The Future of Touch.” Nytimes.org. Accessed November 1, 2015) Hentschläger, Kurt. 2008. http://kurthentschlager.com/ portfolio/zee/zee.html. (Accessed November 1, 2015) Howes, David. 2009. The Sixth Sense Reader. Oxford: Berg Publishers. Howes, David, and Constance Classen. 2013. Ways of Sensing:
THEORIES ON DIGITAL SYNESTHESIA
Understanding the Senses in Society. London: Routledge. Martino, Gail, and Lawrence E. Marks. 2001. “Synesthesia: Strong and weak.” Current Directions in Psychological Science 10.2: 61–65. Iwaki, Akihisa. 2013. “The Body as an Image Processor.” In Diversities in Aesthetics: Selected Papers of the 18th Congress of International Aesthetics, edited by Peng, Gao Jian Ping and Peng Feng. 213–223. Beijing: China Social Sciences Press. Lamontagne, Valerie, et al. 2015. “The Ilinx Garments: Whole Body Tactile Experience in a Multisensorial Art Installation.” Proceedings of the International Symposium on Electronic Art (ISEA), Vancouver, 2015.
64
Schwartz, Robin. 2012. “IBM 5 in 5 2012: Touch.” Accessed November 2, 2015. http://ibmresearchnews. blogspot.ca/2012/12/ibm-5-in-52012-touch.html. Shaw-Miller, Simon. 2013. Eye hEar The Visual in Music. London: Ashgate. Simner, Julia, and Edward M. Hubbard, eds. 2013. Oxford Handbook of Synesthesia. Oxford: Oxford University Press. van Campen, Cretien. 2013. “Synesthesia in the Visual Arts.” In The Oxford Handbook of Synesthesia, edited by Julia Simner and Edward M. Hubbard. 631–646. Oxford: Oxford University Press. Varèse, Edgard, and Chou Wen-Chung. 1966. “The Liberation of Sound.” Perspectives of New Music: 11–19.
Sensing Digital Art: Aesthetic Acts Beyond (the two) Synesthesia(s)
References
Chris Salter
C Ilinx, 2014/2015 (Performance environment) © Chris Salter
Digital Synesthesia: Reminiscing about Science and Art
Romana K. Schuler
Keywords psychophysics, organ projection, physical apparatus, seeing apparatus, experimental physiology and psychology, multimediality, expanded art, expanded cinema, digital art
What do we envision when we think of the new term
everyone is born with these abilities, or whether they
digital synesthesia? What does it mean to attempt to
can be learned, has become a hot topic in memory
integrate the term digital synesthesia into the context
research over the past few decades.
of art and art history? Synesthesia takes on a different meaning when
And yet the question arises as to what digital synesthesia has to do with the original meaning of
combined with the adjective digital. The classic term
synesthesia? Strictly speaking, nothing. Digital stems
synesthesia stems from 19th century conception
from the Latin word meaning digitus, the finger. As of
physiology — as the description for a very specific
the mid-17th century the term is mentioned for the
ability of the sensory organs and sensory perceptions;
first time in connection with fingers (Behrens 2014).
that is the linking of physically separate cognitive do-
The use of the term digital prevailed in connection
mains (fields). How synesthesia develops and whether
with computer technology in the mid-20th century.
65
Today, digital is a buzzword and synonymous with
to expand upon the senses already seems to be upon
modern computer systems. Generally though,
us, to some degree, with the complex computer
the term remains especially vague except for its
systems of today. The goal being aspired towards is
specific use in information technology. In connection
the perfect duplication of the human form. The
with the term synesthesia it points to a current
next step would then be to breathe digitized sensory
problem, which is how can rarely occurring cognitive
perception into this artificial apparatus. From an
processes, that only synesthetitians experience, be
anthropological standpoint, this idea of organ
expressed digitally with digital tools?
projection (Knapp 1877, 27) is age-old and probably
For some time now, it has been clear that
began with the invention of the plow, which was
modern computer science experiments with the
the first complex tool developed by humans
simulation of sensory perception. The potential
(Knapp 1877, 76).
Computer and math: I believe the computer metaphor still has a lot to teach us about the brain. I am particularly interested in brain analogs of operating systems — that are illuminating for cognitive psychology subjects, such as task switching. I also have a long-standing interest in geometry as a constructive theory of perception.
experiments in visual perception (1862 – 1868, Mach). Aside from this fundamental research – the role as pioneer in the study of the human body – his work also led to a mechanical form of expression that appeared to be sufficient for the human body in science. Natural scientists consider the body to
Romana K. Schuler
Sensory Perception: A Mathematical Problem?
be an apparatus, for example an optical apparatus.
(Shabtai Barash 2006)
Even in Freud’s writings we can find that the expres-
The first scientific foundation of the phenomena of
often used.
sion “psychischer Apparat” (mental apparatus) is A main feature of the new sciences in the 19th
can be found in 19th century perception research.
century was the emphasis on the experimental
At that time, the study of human perception did not
method. The experimental procedure became the
generally take place in the physiological areas of
foundational principle of every scientific discipline.
natural sciences but chiefly by physicists and mathe-
The designation of the new specialization (area of
maticians who picked up on the topic and centered
study) was made evident through the title: experi-
their research on it. What was the reason for these
mental physics, experimental physiology and experi-
interdisciplinary tendencies amongst physicists?
mental psychology. This turning point in science was
During the 19th century, physics and mathematics
also actively propagated in society — a succession of
became independent scientific disciplines. Ernst
accomplished scientists held and published so-called
Mach reported that under duress, due to his financial
popular scientific speeches. Beyond that, cultural
situation, he was forced to work in the field of physiol-
changes took place brought on by their far-reaching
ogy after completing his studies in physics because
discoveries, and these resonate to this day. The
he simply could not afford the equipment necessary
insights of the new sciences were picked up and
for studying physics. A second important impulse for
reflected upon with enthusiasm by artists and authors
him was the publication of G.T. Fechner’s Psycho-
alike. In turn, personalities in the sciences such as
physik (1860). Fechner’s approach, as presented in the
Helmholtz or Mach also spoke about visual art in
publication, encouraged Mach to hold a semester-
connection with their new theories on sensory
long lecture on the subject at the University of Graz,
perception. Helmholtz is said to have influenced the
Austria. A short time later he undertook his own
French Impressionists with his optical studies. And
THEORIES ON DIGITAL SYNESTHESIA
66
Digital Synesthesia: Reminiscing about Science and Art
synesthesia and the precursor of digital synesthesia
Mach influenced abstract painting and media artists
were later termed “Chladni’s Klangfiguren” (Chladni’s
such as Dan Graham and Peter Weibel (Schuler 2014).
Figures or Sound Figures). Chladni’s experiments in
Although the previous perception researchers, such as Jan Evangelista Purkinje, David Brewster,
acoustics motivated Michael Faraday, Charles Wheatstone and Ernst Heinrich Weber to lead further
Joseph Plateau, Simon Stampfer, Charles Wheatstone,
experiments in the field. The discovery of sound
Theodor Gustav Fechner, Hermann Helmholtz, Ernst
figures attracted the interest of artists such as Philipp
Brücke and Ernst Mach were physicists or mathe-
Otto Runge, Caspar David Friedrich and Goethe;
maticians, they made considerable contributions to
the latter was also in contact with Chladni. In 1808,
the study of physiological sensory perception. The
Purkinje began to analyze these sound figures in
functions and perceptive capabilities of the human
more detail and in doing so, discovered very similar
body were studied intensively. In this fertile scientific
structures to those of the human eye (Purkinje 1824).
environment the first experiments studying synes-
He shared his discoveries and insights with
thesia took place. The artificial word “syn-ästhesie”
Goethe, among other people. He assumed that the
(syn-aesthesia) was coined in 1866 by the French
visual organ could recognize sound and figures and
neurologist Alfred Volpian. The term was not able
that the eye and ear could possibly match in their
to gain wide acceptance though. In general the
sense of perception.
appearance of this unusual perception was described
Human perception was supposed to be studied
as double or multiple sensations, or as “Farbenhören”
empirically using mathematical formulas and be
(hearing colors) because this was the commonest
experimentally repeatable. A higher-ranking central
form of synesthesia. This phenomenon, as the
research question during the 19th century was
eponym describes, had already been commented
the search for a common canon for the function of
on by many other scientists before him. The oldest
all the senses. Behind the idea was that by means of
account of a synesthetic perception can incidentally
employing a measurable method using repeatable
be found in the Old Testament. When the people
experiments, one would come across a constant.
of Israel were in the desert waiting, the voice of
With the help of this, in an ideal case, mathematical
God could be heard: “... and they saw the voices”
formulas could be deduced. For this purpose, one
(Exodus 20.14).
had to first study the functionality of all the receptors
This genuine synesthesia actually has very little
in terms of touch, taste and sense of smell, and
to do with digital synesthesia. And yet, on the one
especially in terms of the sense of hearing and the
hand genuine synesthesia has begun to be regarded
sense of sight. Each sense was looked at in its
in a decidedly scientific manner. On the other hand
individual components, measured and analyzed.
mathematical, algorithmic or “digital” synesthesia, as we would term it today, was quasi-invented in the 19th century. In this context, special attention should be paid to the very gifted Bohemian-born researcher, Jan
The transition from the scientific perspective of the early-19th-century physical approach to Fechner’s psycho-physics was initiated by Weber’s studies on the sense of touch. Both Weber and Fechner thought they recognized physical-psychological connections
Evangelista Purkinje, who at the turn of the 19th
for the first time. Subsequently, countless conclusions
century grappled with the after image, the kinetic
were drawn as to the skin’s constitution, culminating
movement as well as the phenomena of synesthesia
in a characterology of the skin. Pressure, touch or
among other things (Schuler [2012] 2016). In his
vibrations, the influence of cold and warm, the sense
dissertation dated 1819, Purkinje referred to the
of pain, etc. were studied.
physicist and lawyer Erwin Chladni’s (1787) sound
Fechner recognized a continuation of his own
figures (Fig. 1, 2). Chladni had thin plates strewn with
approach in Weber’s theorem dated 1834. Weber had
sand and set into oscillation with the help of bell-
set weights on his test persons and arbitrarily added
and tuning- forks. In doing so, figures emerged that
more weight in order to ascertain the relationship
67
warnings were sounded as to the one-sidedness of
Fechner was convinced he had found a viable
purely experimental psychology. The well-known
psychophysical fundamental law for all the senses.
Gestalt psychologist Egon Brunswick criticized the
He was able to put it on paper in a mathematical
methodological physicalism that more or less
formula: e = log r. The perception (e) increases
dominated the entire 19th century (Brunswick 1925).
proportionately to the logarithm of the stimulus (r)
At the same time, it was particularly physicists such
(Fechner 1889, 18). This equation became known as
as Mach, representing element-psychology (1886),
the Weber-Fechner law. In his comprehensive
who pointed out the differences between optical and
“Psychophysik” (psychophysics), Fechner was one
geometric similarities. Mach’s work had encouraged
of the first to comment on double- and multiple-
Christian von Ehrenfels in his Gestalt theory (1890)
sensations based on the example of the phenomenon
which was significantly represented in experimental
of hearing colors. The term “Experimentelle
psychology up until the 1940s. It was then superseded
Ästhetik” (experimental aesthetics) can be traced
by cognitive science and cybernetic theory.
Carl Stumpf was a pioneer in the field of experi-
It may well be that all these 19th century trials, deliberations and theories about sensory perception
mental sound- and tonal-cognition in the second
seem outdated. Yet they are the starting point for
half of the 19th century. Initially he was a student of
a field of study that we call digital synesthesia today:
the philosopher and perception researcher Franz
examining and measuring the functions of the
Brentano, and in 1894 Stumpf became professor
senses and sensations so that in the end, they can
of experimental psychology in Berlin. Amongst his
become a graspable algorithm. And so, steering
students were the founders of the famous Berlin
with these machines becomes possible; the machines
Gestalt theory – Max Wertheimer, Wolfgang Köhler,
can be equipped with exceptional features — especially
Kurt Koffka and Kurt Levin.
with regard to big data — namely operating in a
At the age of 22, Stumpf wrote his postdoctoral thesis Die Grundsätze der Mathematik (1870) (The
synesthetic manner. Just like psychophysics, the basis of digital
principles of mathematics) with Hermann Lotz.
synesthesia lies in its double meaning. Synesthesia
A short time thereafter he concentrated on the
implies that the modalities, the perception groups
concept of space and the perception of sound.
of the individual sensory sectors become evident in
He introduced the terms “concordance” and
psychologically different ways.
“discordance” in his Tonpsychologie (1874) (Sound
The wish to rebuild nature in a technological
psychology). Deeply impressed by Weber’s and
manner seems to be an elemental impulse in science.
Fechner’s work, Stumpf began to study inter-
Since the 17th century there has been a desire to
disciplinarily. His greatest achievements can most
dehumanize and reproduce the human spirit in the
notably be found in the field of acoustic perception.
form of machines. At the turn of the 20th century,
He connected these with his phenomenological
machines were used in industry that were equipped
ideas and was able to come to new insights. Around
with simple “Sinneswahrnehmungen” (sensory
1900, with his assistant Erich Hornbostel, Stumpf
perceptions) such as an automatic stop when there
founded the phonogram-archive that is part of the
was a malfunction (Moravec 1990, 22).
Ethnological Museum in Berlin. With his phenomeno-
To date, we remain infected by this history of
logical theory he also influenced Edmund Husserl
thought. Artistic works that evolve in terms of
in the 20th century.
digital synesthesia are tied to this idea.
Only at the turn of the century was the so-called Wissenschaft der Seele (science of the soul) able to be established in psychology. At that time psychology had an experimental orientation. Soon thereafter,
THEORIES ON DIGITAL SYNESTHESIA
Romana K. Schuler
back to Fechner.
68
Digital Synesthesia: Reminiscing about Science and Art
between sensation intensity and stimulus intensity.
Fig. 1
Fig. 1, 2 Chladni’s Sound Figures, 1789 (author’s archive)
Fig. 2
69
Digital Synesthesia and Art Since the beginning of the 20th century psychologists
(Oops we are alive) in his political theater in Berlin. A
have also studied and published on synesthesia to a
four story high stage was created in order to play short
great extent. The psychologist Annelies Argelander
scenes on it as well as film sequences on a screen that
aired her extensive research in the publication Das
had been placed at the center. Piscator worked with
Farbenhören und der synästhetische Faktor der
authors such as Bertold Brecht and Heinrich Mann
Wahrnehmung (Color hearing and the synthetic factor
as well as with artists such as George Grozs, John
of perception) at the Munich Psychology Congress
Heartfield and László Moholy-Nagy on the stage set.
Argelander listed 466 bibliographical references. The writings were printed between 1800 and 1927 and are
their film as “expanding cinema” (Bute and Nemetz
an impressive documentation of the interest in the
1936). Bute, who had come from a painting back-
phenomenon. In 1927 the well-known synesthesia re-
ground, experimented in her animations with synes-
searcher Georg Anschütz organized the first congress
thetic effects in a similar fashion to Oskar Fischinger.
on color-tone-research in Hamburg. An interest in
She used the stroboscope to create effects that she
synesthesia was also reflected in the liberal arts and
called “seeing sound”. In the 1940s the first basic writ-
cultural research from this period. In his speech Die
ings on cybernetics and information theory were pub-
Urgeschichte der Synästhesie und ihre Ausbreitung im
lished. Norbert Wiener’s Cybernetics or Control and
älteren Geistesleben (The prehistory of synesthesia
Communication in the Animal and the Machine came
and its spread in older spiritual life), the Viennese
out in 1948 and C E Shannon published his famous
psychologist Albert Wellek emphasized the relevance
essay The Mathematical Theory of Communication.
of synesthesia in the liberal arts for the first time at a
Wiener was convinced that through technology,
congress on color-tone-studies. One year later Wellek
prostheses for human limbs and sensory organs
completed his doctoral studies with Doppelempfind-
would be developed or their performance would be
ungen und Programmusik (Double sensations and
enhanced. In The Human use of Human Beings (Cy-
program music) (1928). He then moved to Hamburg to
bernetics and Society) from 1950, Wiener refers to art
do research with Anschütz.
and literature as communication media that could not
At the same time, artists became interested in
remain untouched by the up-and-coming communi-
the synesthetic phenomenon. The influence of
cation technology or mass media. In this respect there
synesthetic appearances was represented in musical
was a backlog. He wrote: “History has shown that the
performances (Skrjabin, Lászlo) and in the emerging
development of natural sciences and art generally
medium of film (Ruttmann, Fischinger). In 1915
run parallel to one another. And yet, in the elapsed
Skrjabin, who was a synesthetician, presented his color
half-century, art has evolved far away from science.”
piano to a broad audience in 1915. In the summer
(Wiener 1952, 133) Wiener advocated for an interdis-
of 1925 a concert took place in Kiel (Germany).
ciplinary education; in which artists and intellectuals
Alexander Lásló performed on his further-developed
should not isolate themselves but show more interest
color piano; the instrument was specially built by
in interdisciplinary education (Wiener 1952, 148 – 149).
Ernemann-Werken. His Farblichtmusik (color light
Cybernetics and Information theory actually have
music) was performed about 1000 times in Germany
an effect on artistic work. Whether this is the result of
from 1925 – 27. There were numerous write ups on
a reaction to Wiener’s plea for engineering or for other
the shows and in total more than 40,000 people
reasons is anyone’s guess. While in applied art only
came to experience synesthesia. In 1927 the avant-
color, stone and wood were understood as relevant
garde producer Erwin Piscator directed a complex
material, art forms were being expanded upon with
media stage set for the piece Hoppla wir leben
light (op art) and movement (kinetic art) as well as
THEORIES ON DIGITAL SYNESTHESIA
Romana K. Schuler
In 1936, in an interview, the experimental filmmakers Mary Ellen Bute and Ted Nemetz described
70
Digital Synesthesia: Reminiscing about Science and Art
in 1925. In her 1927 publication with the same title,
sound (sound art). The first computers were rapidly
Kostelanetz who also created the term “mixed-means”
being used. Technical implementation was accepted
(Kostelanetz 1968). In 1964 – 65 the members of the
as artistic media although in the beginning the
American art group USCO (Gerd Stern, Steven Dunkee,
exorbitant costs only allowed for ideas and concepts.
Michael Callahan among others) settled down with
The notion of working with new media began to
their families in an abandoned church in Garnerville
appear in artistic activities. Expanding the concept of
(NY). They were artists, musicians, poets and techini-
the image, and ever more often a complete withdrawal
cians. The basic foci of the group were the theories
from the image, began with the spread of information
of new communication media — Marshall McLuhan
theory and cybernetics. The artist and mathematician
was one of the most important impulse generators.
Benjamin Laposky presented the first computer
During a meeting McLuhan advised the USCO group
drawings in the 1950s. The artist and computer pro-
to “Disregard the content and concentrate upon the
grammer Ken Knowlton developed the computer
effect” (Kuo/Callahan, 2008). In the same interview
program “Beflix” in 1963. With it, he and Stan Van Der
Callahan explained that McLuhan was a scientist and
Beek developed their famous geometric pixel images
that USCO had used science.
such as poem fields at Bell Telephone Labs in New
The influence of consciousness expansion was the
Jersey. The fascination for creating art with a machine
defined goal of the USCO group. According to them,
or with the help of a computer program appeared to
the experimentation with drugs and the use of new
splendidly confirm technology-oriented philosophy
image technology created the psychedelic environ-
as described by Knapp in the 19th century, especially
ment that the members of the group had imagined.
the concept of organ projection.
They and many others understood “psychadelic” as
With the invention of modern computer science,
synonymous with synesthesia. The young movie critic
a breakthrough took place that for the first time
Gene Youngblood, who tried to introduce the term
allowed and enabled the realization of synesthetic
synaesthetic cinema as a new concept of expanded
properties with the help of new technologies. Because
cinema, is an example. Characteristic of synaesthetic
this concept is a de facto algorithm it can be described
cinema was the “simultaneous perception of
as digital synesthesia. This creation of immersive tech-
harmonic opposites” according to the definition
nical environments in order to sound out new limits
(Youngblood 1970, 81).
in perception was tackled by several avant-garde artists
Synaesthetic cinema is neither subjective, nor
at the beginning of the 1960s. Multi-media events
objective, nor non-objective. It is a combination of
solicited a great deal of attention. The architects
everything; what Youngblood calls an extra-objective
Charles and Ray Eames conceptualized a multi-media
expression of reality in new avant-garde film. The
presentation for an American exhibition in Moscow
dual logic of simple opposites such as in/out or up /
in the summer of 1959. On seven massive monitors,
down doesn’t count anymore. Triadic logic is what
seven different films ran at the same time documenting
counts: in/out/maybe. The new synaesthetic cinema
the American lifestyle of the 1950s: A Day in the Life of
corresponds to this type of logic. Cinematic examples
the United States.
are Brakhage’s Dog Star Man, Warhol’s Blue Movie,
From the mid-1960s onward, this new movement in art has been called expanded art or, when referring to film, expanded cinema. Besides these, expressions
Carolee Schneemann’s Fuses und Michael Snow’s Wavelength (1967) (Youngblood 1970, 82–83). Despite all Youngblood’s efforts, the term synaes-
such as “intermedial”, “transmedial” or “mixed-media”
thetic cinema could not prevail against the term ex-
emerged. Such terms as “polymediality” or “poly-
panded cinema. Maybe because Youngblood focused
esthetic” in which the prefix “poly” didn’t mean “many”
on a single medium with his term synaesthetic cinema:
in terms of quantity but rather in terms of multiple
film. Those artists and producers who understood
qualities were already in use. The expression “poly-
their work as expanded cinema no longer understood
artist” was coined in 1968 by the producer Richard
the film screen as a boundary in their artistic work.
71
Jeffrey Shaw and Peter Weibel: From Expanded Cinema to Digital Synesthesia For decades, the artists Jeffrey Shaw and Peter
(Fig. 7). From a higher, more privileged standpoint the
Weibel have studied the domain of perception with
viewer can navigate through an artificial image land-
the help of new media. It therefore seems obvious to
scape via joystick. The digital images can be zoomed
view these artists’ work from an art-historical stance
into to the level of pixel resolution, and verses by Dirk
and in connection with digital synesthesia.
Groeneveld can be heard. The fixed standpoint is reminiscent of a classical perspective in Renaissance
tion of media artists to have worked on expanding the
painting. Instead of perceiving illusory depth percep-
boundaries of perception with expanded cinema since
tion, the viewer can zoom into the individual picture
the mid-1960s, and are associated with it to this day.
points or look at their pixel construction. With legendary works such as Legible City
the experimentation with new technologies; in their
(1988 – 91) or The Golden Calf (1994) Shaw created a
artistic methods and orientation very different
milestone in media art as well as archetypical digital
approaches are noticeable. In Shaw we can see an
synesthetic installations (Fig. 8, 9). Among them are
artistic approach using participatory and narrative
his work in 3D on a 360° projection surface, such
aspects in his work (Fig. 3).
as T Visionarium I and II (2003–2010). The extent to
Weibel, on the other hand, pulls his artistic work
which he is interested in experiments in new narrative
together with his theoretical work — as an epistemic
potential can be seen in his 2014 work Infinite Line
tool broaching the topics of philosophy, sociology,
(in collaboration with Sarah Kenderdine and Edwin
evolution or mathematics (Fig. 4 a-b).
Nadason Thumboo). In a 360° theater, the viewer can
For Shaw the moving image and poetry are central themes. Because the works are conceptualized inactively — the viewer should participate in the
call up 27 poems by Thumboo and create their own personal poem (Fig. 10 a-b). The media artist Weibel developed a series of com-
work — the artist provides an open structure. Similarly
munications concepts in the 1960s that dealt directly
to the artist Graham Stevens (1965), Shaw and Tjebbe
with the expansion of sensory perception. Examples
van Tijen created pneumatic sculptures in urban
are Distanz Sinne (Distance Senses) (1968), the
environments in order to intensify several senses at
Pillen-Project (Pill Project) film (1968) or Information
the same time (Fig 5). “Just as I love the friction
Unit (1967), a razor with multifunctional capabilities
between the real and the virtual, I also love the
such as radio, camera and tv (Fig. 11, 12, 13). Weibel
friction that ensues between a work that is my own
writes on the Pill movie: “I think of films in the form
and that which the audience can claim: it has also
of a new sensory organ that affords me every desired
become partially theirs. I like this type of union with
sensory quality, delivers air to me, whether coming
the audience. Traditional art has so few possibilities
from lung breathing, jam breathing or the reverse,
allowing this to happen and to be experienced”
that does not shy away from discrepancies of sensory
(Shaw 1993, 332).
data, permits me to grasp foam when I see stones,
In the endless filmstrip Continuous Sound and
allows me to smell chocolate when I eat grasses, lets
Image Music (1966) Shaw constructed a chronolog-
me hear violins when I glue envelopes shut, that
ical image perception for the individual sequences
allows me to feel water when I reach for the electrical
of a film (Fig. 6). The movie was shown at numerous
outlet, etc.” (Weibel 1968).
expanded art performances. Between 1985 and 1995 he created the interactive audio visual installation The Narrative Landscape
THEORIES ON DIGITAL SYNESTHESIA
Romana K. Schuler
A noteworthy similarity between the two of them is
Concepts which at that time were considered “too utopian” have become realizable today. For example Die Brille als Welt (Spectacles as World) (1967):
72
Digital Synesthesia: Reminiscing about Science and Art
Shaw and Weibel both belong to the first genera-
The spectacles are comprised of micro-computers in order to program an individual selection of images of the world. Weibel essentially anticipated Zuckerberg’s actual ideas of virtual reality. Weibel understands the act of seeing as an active experience that is not confined to the eyes. “The senses give us models of our surroundings, abstractly expressed theories, hypotheses” (Weibel, 1976, 20). He refers here to the book The Nature of Explanation (1952) written in 1943 by the perception researcher Kenneth Craik (Fig. 14). According to this work, intelligent interfaces of modern image technology should find direct access to the human body, and with that create total independence from space and time, additionally turning off the manipulable interface between image (information) and viewer. Eyes, mouth, ears and limbs, the biological senses interpret nature for us; they are similar to peripheral devices. When they are replaced by tools — by artificial peripheral devices like machines and computers — then a new interpretation of nature occurs. With the help of the term digital synesthesia (or also digital peripherals) the summing-up of new art historical concepts with narrative elements (Shaw) and epistemic ideas (Weibel) takes place. That means for current art history and art theory, the concept of digital synesthesia is as necessary for a no-longernegatable genre as is the question of the origins of digital art.
73
Digital Synesthesia: Reminiscing about Science and Art
THEORIES ON DIGITAL SYNESTHESIA Romana K. Schuler
Fig. 3
Fig. 5
74
Fig. 4a
Fig. 3 Jeffrey Shaw, Neil Brown, Dennis Del Favero, Matthew McGinity, Peter Weibel, T_Visionarium II, 2008 archive Jeffrey Shaw
Fig. 4a, 4b Peter Weibel, Life in the 20th Century | 222 Million Murders, 2011 Photo: Jens Barth Fig. 4b Fig. 5 Jeffrey Shaw, Tjebbe van Tijen, Sean Wellesley-Miller, Alpevent, Bern 1968 archive Jeffrey Shaw
75
Digital Synesthesia: Reminiscing about Science and Art
THEORIES ON DIGITAL SYNESTHESIA Romana K. Schuler
Fig. 6
Fig. 7
76
Fig. 8
Fig. 6 Jeffrey Shaw,Tjebbe van Tijen, Willem Breuhen (sound), Continous Sound and Image Music, 1966 archive Jeffrey Shaw
Fig. 7 Jeffrey Shaw, Dirk Groeneveld, The Narrative Landscape, 1985/1995 Photo: Oscar van Alphen
Fig. 8 Jeffrey Shaw, Legible City, 1988 – 91 Photo: Marco Caselli
Fig. 9
Fig. 9 Jeffrey Shaw, The Golden Calf (1994) archive Jeffrey Shaw
77
Fig. 10a, 10b Jeffrey Shaw, Sarah Kenderdine, Edwin Nadason Thumboo, The infinite Line, 2014 (archive Jeffrey Shaw)
Fig. 11 Peter Weibel, Distanzsinne, 1968 (archive Peter Weibel)
Romana K. Schuler
Fig. 12 Peter Weibel, Pillenfilm, 1968 (archive Peter Weibel)
Fig. 13 Peter Weibel, Information Unit, 1967 (archive Peter Weibel)
Fig. 14 Peter Weibel, The Nature of Explanation, 1973 (archive Peter Weibel)
Digital Synesthesia: Reminiscing about Science and Art
Fig. 10a
Fig. 10b
THEORIES ON DIGITAL SYNESTHESIA
78
Fig. 11
Fig. 12
Fig. 13
Fig. 14
79
Argelander, Annelies, 1927, Das Farbenhören und der synästhetische Faktor der Wahrnehmung, Jena: Gustav Fischer. Barash, Shabtai, 2006, “The mindbody problem: neurophysiology of looking and seeing”, in Life Science 2006, Rehovot: Weizmann Institute of Science. Behrens, Roger, 2014, “Digitale Frist. Computer. Pop (Beta Vision)”, in Bug Report Digital war besser, Mainz: Testcard 24, 42– 45. Brunswik, Egon, 1922, “Prinzipienfragen der Gestalttheorie”, in Beiträge zur Problemgeschichte der Psychologie. Festschrift zu Karl Bühler’s 50. Geburtstag, Jena: Verlag Gustav Fischer, 95 – 97. Chladni, Ernst F. Friedrich, 1787, Entdeckung über die Theorie des Klanges, Leipzig: Weidmanns Erben und Reich.
Knapp, Ernst, 1978, Grundlinien einer Philosophie der Technik [1877], Düsseldorf: Stern-Verlag, Janssen & Co. Kostelanetz, Richard, 1980, The Theater of Mixed-Means. An Introduction to Happenings, Kinetic Environments and Other Mixed-Means Presentations [1968], New York: RK Editions. Kuo Michelle, 2008, Special Effects. Michelle Kuo speaks with Micheal Challahan about USCO, (www.questia.com/magazine). Mach, Ernst, 1861, “Über das Sehen von Lagen und Winkeln durch die Bewegung des Auges. Ein Beitrag zur Psychophysik”, in Wiener Sitzungsberichte, Classe II, vol. 43, Vienna, 215 –224. Mach, Ernst, 1865, “Über die Wirkung der räumlichen Vertheilung des Lichtreizes auf die Netzhaut“, in Wiener Sitzungsberichte, Abt. II, vol. 52, Vienna 303 –322.
Purkinje, Jan, 1979, “Abhandlung über die physiologische Untersuchung des Sehorgans und des Hautsystemes“ [dissertation Breslau 1823], reprint in Acta Historica Leopoldina, nr 11, Halle/Saale: Deutsche Akademie der Naturforscher Leopoldina.
Stumpf, Carl, 1955, Über den Psychologischen Ursprung der Raumvorstellung [18, Amsterdam: E. J. Bonset.
Rösch, Gabriele, 2007, “Albert Wellek: Synästhesie als kulturbildendes Phänomen“, in Ausdruck Ausstrahlung Aura, Synästhesien der Beseelung im Medienzeitalter, K. Clausberg, E. Bisanz, C. Weiller eds., Bad Honnef: Hippocampus Verlag, 13 –28.
Weibel Peter, 1976, “Kommentare zur Natur der Technik”, in Konrad Balder Schäuffelen, sprache ist fuer wahr ein koerper, München: Städtische Galerie im Lenbachhaus, 19 –24.
Schuler, Romana Karla, 2014, “The Experiments of Perception in Science and Art by Ernst Mach, Dan Graham and Peter Weibel” in Art Theory as Visual Epistemology, Harald Klinke ed., Newcastle upton Tyne: Cambrigde Scholars Publishing, 125 –144. Schuler, Romana Karla, 2016, Seeing Motion. A History of Visual Perception in Art and Science, Berlin, Boston: De Gruyter. Shaw, Jeffrey, 1993, “Reisen in virtuelle Realitäten”, in F. Rötzer, P. Weibel, eds., Cyberspace. Zum medialen Gesamtkunstwerk, Munich: Boer Verlag.
Weibel, Peter, 1968, “Pillen-FilmProjekt“, in Supervisuell nr 2, Zürich.
Wiener, Nobert, 1952, Mensch und Mensch-Maschine, Berlin, Frankfurt a. Main: Alfred Metzner Verlag. Youngblood, Gene, 1970, Expanded Cinema, New York: E. P. Dutton & Co., Inc.
Digital Synesthesia: Reminiscing about Science and Art
Craik, Kenneth, 1952, The Nature of Explanation [1943], Cambridge: University Press
Fechner, G. Theodor, 1860, Elemente der Psychophysik, Leipzig: Breitkopf & Härtel.
Romana K. Schuler
References
THEORIES ON DIGITAL SYNESTHESIA
80
The Incidental Synesthete
Jeffrey Shaw / Sarah Kenderdine
Keywords multi-sensory, prosthetic, kinesthetic, embodiment, virtuality, panoramic, embodiment, place, nature, manufacture, ataxia
Sensory experiences are at the forefront of cultural
Embodiment offers us a strategy to revive the “synes-
analysis today, overturning linguistic and textual
thetic project” in the trajectory of the post-human and
analysis and supporting both experiential and phe-
hyper-virtual, and allow it to cohere with the promise
nomenological inquiry. Embodiment theories attempt
of “the art to come.” “Synesthetic art historically refers
to understand the mind as a set of physical processes
to multi-sensory experiments in the genres of visual
derived throughout the brain and body of a human,
music, music visualization, audiovisual art, abstract
which ultimately serve his or her action in the physical
film, and intermedia. Distinct from neuroscience, the
world. Embodiment is multisensory and results from
concept of synesthesia in the arts is regarded as the
effects of visual, auditory, tactile, olfactory, and
simultaneous perception of multiple stimuli in one
gustatory cues; it is an immersive entanglement
gestalt experience” (Wikipedia). These “multi-sensory”
through and with context and environment. The body
experiments were a hallmark of art practices that
carries time into the experience of place, and land-
emerged in the 50s and 60s, and were an ineluctable
scape and past and present fold in upon each other.
inspiration for artists who were looking for new
81
avenues of expression and experience. For example,
inflatable “eventstructures” (Waterwalk 1969) and
earlier endeavors in “visual music” such as Alexander
“expanded cinema situations” (Corpocinema 1967)
Scriabin’s color organ Chromola were given new life
which were manifestly directed at incorporating the
in the liquid light shows that surfaced in the early
viewer’s body as being integral to the expression and
1960s as an accompaniment to electronic music and
operation of the artwork. and Sarah Kenderdine’s recent joint practice, albeit
music. Furthermore, the artists Mark Boyle and Joan
as a concomitant to research that focuses on strat-
Hills took the legacy of Piero Manzoni’s seminal
egies of immersive interactive visualization. In this
explorations of the conjunction between art
respect we can single out one particular installa-
production and human production (Merda d’artista,
tion, Place-HAMPI (2006), to elucidate the dynamic
Fiato d’artista) to directly infuse these liquid light
“multi-sensory” conjunction that emerges from the
shows with their own bodily fluids.
conjunction of 3D panoramic visualization with the
In the installation Black Airground (1968) at the
performative engagement of an audience, which is
Oxford Museum of Modern Art, Jeffrey Shaw used
actualized by the unique design of a technological
a first generation sound reactive light controller to
apparatus. Assuming the reader is already familiar with
modulate three 1000-watt light bulbs at various
this installation (http://www.place-hampi.museum/)
frequencies of sound input, so that the illumination
we focus on a visitor survey done by Anita Kocsis
of the exhibition space was entirely dependent on
during its exhibition at the Martin Gropius-Bau in
vocalization from the visitors. This was an expression
Berlin in 2008. In this particular questionnaire, the
of a further and more fundamental development of
visitors were offered a simple outline drawing of
the contemporary “multi-sensory” experience
a human figure and asked to put marks on it in
whereby “interactivity” and the engagement of the
response to the question “where on your body did
viewer via various technological prostheses created a
you experience this work?”
dynamic conjunction of between the artwork and the
Figs. 1–6 show a selection of the responses that
bodily action of the viewer. A seminal work in this
were received. Fig. 7 is a composite drawing of all
respect was Julio Le Parc’s Glasses for a Different
visitor responses, which substantiate the notion
Vision (1965), which proposed the optical prosthesis
of embodiment as being at the heart of the multi-
as an interactive apparatus to re-view the world, and
sensory experiment.
thereby prefigures the now-emerging ubiquity of VR goggles such as the Oculus Rift and Google Glass.
In his book Body and Image: Explorations in Land-
And even more so than “virtual reality”, it is “augment-
scape Phenomenology (2008), Christopher Tilley
ed reality” (the audio and/or visual conjunction of real
contrasts iconographic approaches to the study of
and virtual elements) that signals one of the most
representation with a perspective based on kinesthetic
compulsive and eloquent expressions of “the simulta-
enquiry. “Iconographic approaches … grant the
neous perception of multiple stimuli in one gestalt
primacy to the human mind as a producer of the
experience” in our time. Jeffrey Shaw explored the
meaning of the images through sensory perception.
generative capacity of AR in numerous artworks
It is the mind that responds in a disembodied way.
including Virtual Sculpture (1981), Inventer La Terre
Kinesthetic approaches, by contrast, stress the role
(1986), and The Golden Calf (1994), and underlying
of the carnal human body. The general claim is that
each of these works was the principle of the viewer’s
the manner in which we perceive, and therefore
embodied engagement as a necessary precondition
relate to visual imagery, is fundamentally related to
for a synesthetic conjunction of perception and
the kinds of bodies we have. The body both limits
experience. That precondition also drove the
and constrains and enables us to perceive and react
formulation of his earlier audience participation
to imagery in specific embodied ways.”
THEORIES ON DIGITAL SYNESTHESIA
82
Jeffrey Shaw / Sarah Kenderdine
adapted for performances of rock or psychedelic
This precondition remains central to Jeffrey Shaw
The Incidental Synesthete
avant-garde theater performances and were later
One could say that IN_SIDE VIEW is an allegory for
conflagration, form and dissolution, purpose and
the multi-sensory encounter between the body
affliction, past and present, biology and machine,
and the perceptual imaginary, where the imaginary
the gratuitous and the necessary, between survival
itself embodies a further encounter between the
and ataxia. Nature-society is seen as a complex
natural and made world. In this respect it contributes
entanglement of people-things, here played out
to the paradoxical connection between the concepts
in an embodied theater of hybrid realities induced
of nature and culture proposed by Vilem Flusser
through immersive display systems. IN_SIDE VIEW
(Natural:Mind, 1967), by setting up an assembly of
concedes with a pataphysical riposte to these
confrontations and concurrences – amongst others
technological affordances that now constitute con-
between tongue and apparatus, between mouth and
temporaneity’s diametric spaces of phenomeno-
eye, between stone and vegetation, steel and
logical encounter.
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Figs. 1–6 These are a selection of the responses that were received.
83
Jeffrey Shaw / Sarah Kenderdine The Incidental Synesthete
Fig. 7 Co-Designing New Media Spaces, Anita Kocsis, 2007
THEORIES ON DIGITAL SYNESTHESIA
84
The Interaction between Seeing and Moving: Eye Movements as a Window to the Mind ? Miriam Spering
Keywords visual perception, eye movements, brain, cognition, visual neuroscience, active vision, perception-action dissociation, awareness
Seeing and perceiving the visual world is an active
Montagnini, 2011). Smooth pursuit eye movements
and often multimodal process that involves orienting
are finely tuned to the visual object’s properties
eyes, head and body towards an object of interest.
of motion, direction and speed, and help to extract
It is also a highly dynamic process during which the
visual motion information to enable more accurate
eyes continuously scan the visual environment to
and precise interactions with our environment
sample information. When we view the world around
(Spering et al. 2011). Furthermore, we have the
us, our eyes are always in motion. We move our gaze
ability to hold our gaze steadily on stationary objects
to fixate on objects of interest using fast eye move-
of interest using fixational eye movements, and to
ments called saccades. These small, highly accurate
compensate for movements of our head and body
movements occur at high speed and frequency – up
using the vestibular-ocular reflex. Failure to stabilize
to three times per second, more frequently than a
gaze or to move gaze to objects of interest results
heartbeat – during activities such as reading or scene
in motion blur, loss of visual acuity and impaired
viewing that require the perception of fine spatial
fine spatial vision. It thus seems that the link between
detail (Carpenter, 1988; Kowler, 2011). We also track
eye movements and visual perception is tight: we
objects with our eyes to keep them close to the
move our eyes in order to enhance and optimize
fovea – the area in the eyes producing the highest
the perception of visual objects and events in our
visual acuity – using a type of eye movement
environment, a process sometimes referred to as
termed smooth pursuit (Lisberger, 2010; Spering &
active vision (Findlay & Gilchrist, 2008).
85
regular and reproducible, and that a viewer’s eyes fol-
by many research disciplines, ranging from develop-
low a similar path when viewing an image repeatedly,
mental and cognitive psychology to computer science
reflecting how objects or images are remembered
and art history, to measure visual perception, object
and recognized (Noton & Stark, 1971). The discovery
categorization, recognition, and many other mental
that cognitive processes influence eye movements
processes. Eye movements are also utilized to design
was first made by Russian physicist and vision scientist
and assess products and applications (e.g., in web-
Alfred Yarbus, sometimes considered the founder of
page design) or to evaluate performance (e.g., in a
modern eye movement research (Yarbus, 1967; Tatler
driving simulator or in sports). These studies and
et al. 2010). Yarbus made several key observations:
applications are based on the fundamental assump-
first, when viewing a painted scene, the initial
tion that eye movements reflect what we see, perceive,
fixation is preferentially directed at persons within
think or desire.
the scene, and more specifically, to the individuals’
A rich literature in experimental psychology indi-
faces (Buswell, 1935; Fletcher-Watson et al. 2008)
cates that eye movements reflect awareness – the
and their eyes (Kingstone, 2009). Second, during
ability to explicitly report something – and can
longer viewing, gaze follows a cyclical viewing
serve as a key to our understanding of visual and
pattern and repeatedly revisits the most important
cognitive processes. For example, voluntary eye
elements of a picture. Third, gaze patterns depend
movements such as saccades and smooth pursuit
on instruction – a finding we have reproduced here
are driven by visual information (Kowler, 2011;
(Fig. 1; see also Castelhano, Mack & Henderson,
Spering & Montagnini, 2011) and their dynamic
2009; Tatler et al. 2010). In more recent years, eye
properties sensitively reflect how our brain processes
movement responses have been formalized to
visual information (Lisberger, 2010). When we look
investigate cognitive processes during visual search,
at an object or image, our gaze and the focus of our
reading, and scene viewing (for reviews, see Rayner,
attention are quickly attracted to locations of high
1988; 2009). Research in vision sciences has focused
salience (Itti & Koch, 2001) – conspicuous areas in
on the role of eye movements for visual perception.
the image that stand out from their context because
Many have shown that eye movements enhance
of their visual properties. Fig. 1a shows a photo of the
different aspects of vision; for instance, the ability to
Vancouver north shore. When asked to simply look
perceive fine spatial detail or to extract information
at this image, a viewer’s gaze is attracted by salient
about an object’s motion, direction and speed
objects or locations, such as the black birds in front
(for reviews, see Kowler 2011; Schütz, Braun &
of the blue sky, the red ships surrounded by a blue-
Gegenfurtner, 2011; Spering & Montagnini 2011).
grey context and the mountain peaks (Fig. 1b).
Eye movements are also critical in guiding visual
Interestingly, gaze patterns may also reflect a specific
processes during everyday activities that involve hand
cognitive task. When asked how many birds were in
and body movements, such as making a cup of tea
the photo, our viewer systematically scans water and
or playing sports (for reviews, see Hayhoe & Ballard,
rocks, and their gaze lingers on an ambiguous area
2005; Land & Tatler, 2009).
to the far left, where a seagull can be seen (Fig. 1c). We produced these eye movement patterns by showing the image only once, for 10 seconds. However, previous studies indicate that eye movement patterns during object or scene viewing are highly
THEORIES ON DIGITAL SYNESTHESIA
86
The Interaction between Seeing and Moving: Eye Movements as a Window to the Mind?
Eye movements are used in many contexts and
Miriam Spering
Eye Movements as a Window to the Mind
When Eye Movements Reflect the Processing of Unaware Information Recently, however, the established view of there being
throw and catch (Fig. 3a); in another condition, the
a tight link between visual perception and eye move-
movie showed the actor merely pretending to throw
ments has been questioned and several studies have
and catch a ball by making appropriate hand move-
revealed situations in which eye movements reflect
ments and following the imaginary ball trajectory with
the processing of visual information that the viewer
head and eyes (Fig. 3b). Most observers perceived the
is not aware of (for a review, see Spering & Carrasco,
illusion and also claimed that their eyes followed the
2015). We developed an experimental paradigm in
ball. Interestingly, their gaze was not fooled by the
which one part of the visual scene is suppressed from
illusion: while they perceived the ball to have been
awareness and not perceived, using an established
thrown, their eyes mostly remained fixated on or
perceptual procedure called binocular rivalry flash
around the actor’s head.
suppression. During binocular rivalry, different images
Many other studies have reported quantitative
are presented to the two eyes using mechanical
differences between perception and eye movements,
devices such as a stereoscope. Instead of merging
in which eye movements are more sensitive than our
the two images into one coherent percept, viewer
conscious perception. For example, the eyes may
perception usually alternates between the two images
sensitively react to small differences in the speed of
(Blake & Logothetis, 2002), indicating that one of the
a moving target that go completely unnoticed
two images is selectively suppressed from awareness.
(Tavassoli & Ringach, 2010).
We showed that the suppressed image can nevertheless influence eye movements (Spering, Pomplun &
This series of examples (reviewed in Spering & Carrasco 2015) reveals that eye movements may be
Carrasco, 2011). Our observers were shown two
more sensitive than visual perception and can some-
images on a computer monitor, viewed through a
times even reflect visual information that we are not
stereoscope, which projected the image on the left
aware of. Eye movements may be earlier, faster
side of the screen to the left eye only, and the image
and more accurate predictors of the way we locate
on the right side to the right eye only (Fig. 2a). One
and track events in the visual world compared
image showed a pattern moving to the right, the
with perceptual reports. Eye movements may thus
other image showed a pattern moving downwards.
be more than just a window to the mind: they
Congruent with research findings in binocular rivalry,
may serve as sensitive indicators of unconscious
observers reported seeing only one of the two images
visual processing.
at a time, either in a rightward or downward motion (example of downward shown in Fig. 2b). Remarkably, the eyes responded sensitively to the unperceived information as if it was completely visible, and moved diagonally down and right (Fig. 2c), to a motion direction that was almost never perceived. This study shows that we can perceive one thing while our eyes actively look at or track another object or part of a visual scene. Another impressive example comes from a study by Kuhn and Land (2006), in which observers were asked to view movies of an actor throwing and catching a ball. In one condition, the movie clip showed an actual
87
How Does the Brain Control Eye Movements without Awareness? One of the key questions is how these findings
than perception, can provide further insight into
can be reconciled with decades of research showing
brain function. We recently proposed that different
tight links between perception and eye movement.
brain pathways may mediate situations in which
The idea that perception and movement may be
eye movements reflect awareness and closely
processed differently in the brain is well-established
match perception versus situations in which eye
in the neurosciences: the dual-pathway model
movements reflect visual processing without
(Goodale & Milner, 1992; 2013) proposes largely
awareness (Fig. 4).
perception and vision-for-action, which may control
physiological, neuroimaging and clinical studies
movements without awareness (Milner, 2012;
now indicates the involvement of a subcortical
Goodale & Milner, 2013). Our novel findings,
pathway (analogous to the one shown in blue in
indicating that eye movements are more sensitive
Fig. 4) in unconscious visual processing.
Conclusions Visual perception is an active process that relies on
and more sensitive to visual information that we do
fast and accurate eye movements. The tight per-
not consciously perceive. Such perception-movement
ception-movement link can be utilized to infer the
dissociations may be adaptive responses to different
workings of the mind by evaluating the spatiotemporal
task requirements and may allow insight into visual
dynamics of eye movements. However in some situ-
processing without awareness and its underlying
ations eye movements may be faster, more accurate
brain mechanisms.
Fig. 1a
THEORIES ON DIGITAL SYNESTHESIA
Fig. 1b
Fig. 1c
88
Miriam Spering
Converging evidence from behavioral, neuro-
The Interaction between Seeing and Moving: Eye Movements as a Window to the Mind?
independent brain processing for vision-for-
Fig. 1 (a) Vancouver north shore mountains. (b) One viewer’s gaze pattern during 10 seconds of free viewing; the size of the red circles indicates fixation duration; red lines indicate saccades. (c) Gaze pattern when the same viewer counted the number of birds shown in the photo.
Fig. 2 (a) Experimental set-up showing two visual images on a computer monitor, viewed through a stereoscope (not shown) so that each image is seen by one eye only; in this example, the left eye viewed the rightward moving image, the right eye viewed the downward moving image. (b) Perception follows one motion direction exclusively. The red arrow represents mean perceived motion direction across eight observers. (c) Eye movements track the average motion direction of both images (pattern motion) – a direction that is perceived in less than 1 % of all trials. Adapted from Spering and Carrasco (2015).
Fig. 3 (a) A ball is thrown up in the air and caught again; observers’ gaze (red circle marks center of fixation) follows the ball. (b) Actor pretends to throw and catch the ball; observers’ gaze (red circle) is centered on or around the actor’s head. Images and schematic data points created following Kuhn and Land (2006).
Fig. 4 Schematic brain pathways for perception and eye movements in a human brain. The red pathway mediates visual information for perception (retinogeniculate pathway); the blue pathway mediates visual information for eye movements in the absence of awareness. This fast, retinocollicular pathway directly connects the eyes with brainstem structures and bypasses some of the structures involved in the retinogeniculate pathway. Adapted from Spering and Carrasco (2015).
Fig. 2a
Fig. 2b
Fig. 3a
Fig. 2c
Fig. 3b
Fig. 4
89
Research partly funded by a Natural Sciences and Engineering Research Council Canada (NSERC) Discovery Grant and the Peter Wall Institute for Advanced Studies at UBC.
References Blake, R. & Logothetis, N. (2002). Visual competition. Nature Reviews Neuroscience, 3, 1–11. Buswell, G. T. (1935). How people look at pictures: a study of the psychology of perception in art. Chicago, IL: University of Chicago Press. Carpenter, R. H. S. (1988). Movements of the eyes. London, UK: Pion Publishers. Castelhano, M. S., Mack, M. L., & Henderson, J. M. (2009). Viewing task influences eye movement control during active scene perception. Journal of Vision, 9(3):6, 1–15.
Goodale, M. A. & Milner, A. D. (1992). Separate visual pathways for perception and action. Trends in Neurosciences, 15, 20–25. Goodale, M. A. & Milner, A. D. (2013). Sight unseen: an exploration of conscious and unconscious vision. Oxford, UK: Oxford University Press. Hayhoe, M. & Ballard, D. (2005). Eye movements in natural behavior. Trends in Cognitive Sciences, 9, 188–194. Itti, L. & Koch, C. (2001). Computational modelling of visual attention. Nature Reviews Neuroscience, 2, 1–11. Kingstone, A. (2009). Taking a real look at social attention. Current Opinion in Neurobiology, 19, 52–56. Kowler, E. (2011). Eye movements: the past 25 years. Vision Research, 51, 1457–1483. Kuhn, G. & Land, M. F. (2006). There’s more to magic than meets the eye. Current Biology, 16, R950–R951. Land, M. F. & Tatler, B. W. (2009). Looking and acting: vision and eye
movements in natural behaviour. Oxford, UK: Oxford University Press.
pursuit eye movements: A review. Vision Research, 51, 836–852.
Lisberger, S. G. (2010). Visual guidance of smooth-pursuit eye movements: sensation, action, and what happens in between. Neuron, 66, 477–491.
Spering, M., Schütz, A. C., Braun, D. I., & Gegenfurtner, K. R. (2011). Keep your eyes on the ball: smooth pursuit eye movements enhance prediction of visual motion. Journal of Neurophysiology, 105, 1756–1767.
Milner, A.D. (2012). Is visual processing in the dorsal stream accessible to consciousness? Proceedings of the Royal Society B, 279, 2289–2298. Noton, D. & Stark, L. (1971). Scanpaths in eye movements during pattern perception. Science, 171, 308–311. Rayner, K. (1988). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372–422. Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology, 62, 1457–1506. Schütz, A. C., Braun, D. I., & Gegenfurtner, K. R. (2011). Eye movements and perception: A selective review. Journal of Vision, 11(5):9, 1–30.
Spering, M. & Carrasco, M. (2015). Acting without seeing: eye movements reveal visual processing without awareness. Trends in Neurosciences, 38, 189–258. Tatler, B. W., Wade, N. J., Kwan, H., Findlay, J. M., & Velichkovsky, B. M. (2010). Yarbus, eye movements, and vision. Iperception, 1, 7–27. Tavassoli, A. & Ringach, D. (2010). When your eyes see more than you do. Current Biology, 20, R83–R94. Yarbus, A. L. (1967). Eye movements and vision. New York, NY: Plenum Press.
Spering, M. & Montagnini, A. (2011). Do we track what we see? Common versus independent processing for motion perception and smooth
The Interaction between Seeing and Moving: Eye Movements as a Window to the Mind?
Findlay, J. M. & Gilchrist, I. D. (2008). Active vision: the psychology of looking and seeing. Oxford, UK: Oxford University Press.
Fletcher-Watson, S., Findlay, J. M., Leekam, S. R., & Benson, V. (2008). Rapid detection of person information in a naturalistic scene. Perception, 37, 571–583.
Miriam Spering
Acknowledgements
THEORIES ON DIGITAL SYNESTHESIA
90
Digital Synesthesia
Peter Weibel
Keywords computability, correlation, data, database, entropy, information, music, noise, randomness, undecidability
Analog synesthesia describes the processing of
functions. Evidently the visual cortex of the brain can
sensory data in the human brain. Analog synesthesia
also process auditory stimuli, likewise the area of
is the effect of an interior experience of a subject.
the cortex responsible for acoustic stimuli can also
The scientific explanation at the moment is the
process visual stimuli.
following: 1. A human being has several doors of perception
4. Due to this neuroplasticity of the brain, a cortical remapping can take place. As a consequence visual
like the eyes, the ears, etc. Through these sensory
stimuli can produce auditory stimuli and acoustic
organs light waves or sound waves, so-called
stimuli can produce visual stimuli.
stimuli, enter the brain. 2. The brain consists of different areas in which
These four steps describe the classical synesthetic experience, which around 1900 evoked great interest
these different sensory stimuli are processed.
and produced a lot of relevant art works, from
According to the classical location theory, different
Alexander Scriabin to Arnold Schönberg, from Wassily
areas of the brain are in charge of different
Kandinsky to Mikalojus Konstantinas Čiurlionis, i.e.,
sensory stimuli.
from synesthetic musicians to synesthetic painters.
3. But observation and experience show that neighboring areas of the brain can adopt different
Digital synesthesia is not a subjective experience, but an objective observable phenomenon. It does
91
not happen in the human brain and it has nothing to
repeated both in the same, and in several Papers, I
do with the processing of human sensory stimuli.
found that the Observations agreed well enough with
It happens in the physical world according to the laws
one another, and that the Rectilinear Sides MG and FA
of nature. Digital synesthesia does not connect and
were by the said cross Lines divided after the manner
intertwine sensory modalities, but operates within
of a Musical Chord. Let GM be produced to X, that MX
the physical properties of the world.
may be equal to GM, and conceive GX, λX, ιX, ηX, εX,
γX, αX, MX, to be in proportion to one another, as
At the beginning of digital synesthesia is Isaac Newton‘s Opticks, or a Treatise of the Reflexions,
the Numbers, 1, 8/9, 5/6, 3/4, 2/3, 3/5, 9/16, 1/2, and
Refractions, Inflexions and Colours of Light (1704). In
so to represent the Chords of the Key, and of a Tone,
his book, Newton summarizes the conclusions from
a third Minor, a fourth, a fifth, a sixth Major, a seventh and an eighth above that Key: And the Intervals Mα, αγ,
his discovery that white light consists of rays that
γε, εη, ηι, ιλ, and λG, will be the Spaces which the
refract differently and correspond to the color spectrum. Newton invented the art of systematic
several Colours (red, orange, yellow, green, blue,
experimentation as to be able to develop a testable
indigo, violet) take up.” 4 So he was the first to provide
theory. He therefore starts his book on optics with
a mathematical theory on the relationship between
the following sentence: “My Design in this Book is
colors and sounds.
not to explain the Properties of Light by Hypotheses,
This is precisely the origin of digital synesthesia:
but to propose and prove them by Reason and
The relationship between visual and acoustic
Experiments.” We can hear Roger Bacon‘s (Opus
phenomena in terms of numbers and digits. As we
majus, 1266) words echoing here. The British
know, the Greeks had already described music as a
Franciscan monk and philosopher Roger Bacon
form of mathematics. But it was Newton who was also
1
able to describe color theory as a form of mathematics.
Newton was one of the first to see a parallelism between the world of light and the world of sound. Newton not only investigated optical laws
When music and light are just two different forms of mathematics then it is just consequent to link them objectively in a form of coefficient theory. Hermann von Helmholtz further expanded on
but also saw how these were related to the laws
the mathematical theory of acoustic and optical
of acoustics: “For as Sound in a Bell or musical String,
stimuli in his physiology of hearing and seeing, Die
or other sounding Body, is nothing but a trembling
Lehre von den Tonempfindungen als physiologische
Motion, and in the Air nothing but that Motion
Grundlage für die Theorie der Musik (1863). 5 With
propagated from the Object, and in the Sensorium ’tis
his mathematical studies, Helmholtz pinpointed new
a Sense of that Motion under the Form of Sound; so
links between physics, physiology, and aesthetics,
Colours in the Object are nothing but a Disposition
as Gustav Theodor Fechner did with his theory of
to reflect this or that sort of Rays more copiously than
psychophysics (Elemente der Psychophysik, 1860). 6
the rest; in the Rays they are nothing but their Disposi-
After Newton, James Clerk Maxwell was the
tions to propagate this or that Motion into the
first to provide a mathematical representation of
Sensorium, and in the Sensorium they are Sensa-
electromagnetic waves, the carrier of light (color)
tions of those Motions under the Forms of Colours.” 3
and sound. Maxwell‘s equations are a set of partial
Newton described sound and light as propagation of
differential equations that form the foundation of
motion. He described music and color as scientific
classical electrodynamics, classical optics, and
objects. Digital synesthesia is a scientific technical
electric circuits. Maxwell‘s equations describe how
operation. The most important discovery was that
electric and magnetic fields are generated and altered
Newton also devised a kind of coefficient theory
by each other.
between sounds and colors that he defines numerically: “And this Operation being divers times
THEORIES ON DIGITAL SYNESTHESIA
Peter Weibel
knowledge, namely, by reasoning and experience.”
2
Between 1861 and 1864 Maxwell devised an electromagnetic theory of light in which he first asserted the
92
Digital Synesthesia
wrote: “For there are two modes of acquiring
existence of electromagnetic waves and then equated
floats and convulses in darkness to a stark and powerful,
the speed of light with the speed of electromagnetic
highly synchronized soundtrack. The velocity of the
waves. Heinrich Hertz, a student of Hermann von
moving images is ultra-fast, some hundreds of frames
Helmholtz, conducted between 1886 and 1888
per second, so that the work is a performance test for
thought experiments and later lab experiments in
the audio and visual devices, as well as a response test
Karlsruhe with sparks and vibrations, enabling him
for the audience’s perceptions, pushing it to its limits.
7
to identify electromagnetic waves and light waves.
As we can learn from this example digital
Radio, TV, radar, and cordless phones have become
synesthesia introduces a new concept of music and
the results of this wireless communication of sounds
image. Musical and visual systems are now positioned
and images by electromagnetic waves. Hertz is thus
between entropy and information, noise and order.
considered one of the originators of physical vibra-
In the area of digital synesthesia the artist operates
tion theory, which applies to both optic and acoustic
with algorithms, bits, mathematical precision, e.g., the
waves. The unit for the frequency at which these
logarithm of the number of possible options. The
occurrences are repeated per second in a periodic
classical correlation between optics and sound now
signal was named after Heinrich Hertz in 1930 and
has a new basis: The correlation between information
introduced in 1935 as Hz. Despite the definition, the
and entropy.
use of the unit is not restricted to periodic vibrations.
From Newton’s Experimentum Crucis 9, in which
Regularly repeated events, such as the frequency at
white light is shown to be the sum of all prismatic
which a computer creates backup copies of files or
colors, we can learn that noise may be the optimal
issues commands can also be measured in Hz
state of music, the summa maxima of all melodies.
(frequency). Modern processors have a frequency
Noise could be the white light, containing all colors
in the gigahertz (GHz) range, meaning a billion
and sounds. The experimentum crucis of digital
cycles a second.
synesthesia is therefore the following: Defining music
Many contemporary sound experiments are based
as entropy and information in digital systems marks
on these transversal and longitudinal vibrations. In
the beginning of a digital universe of sounds in
their collaborative research project, cyclo. id (2000),
which each instrument can emulate the sound of any
which focuses on the visualization of sound, the artists
other instrument and each object can simulate the
Ryoji Ikeda and Carsten Nicolai visually register each
sound of any other object.
wave form created by different frequencies. Approxi-
In the analog world there was a correlation
mately more than a thousand waveforms are depicted
between sensory organ and sensory stimuli. The eye
establishing a kind of virtual Hertzian library of sound.8
was the organ to process light waves, a very small
The artists developed a database of sounds, using
spectrum of the electromagnetic waves radiated by
equipment developed originally for phase correlation
the sun. In a certain sense the eye is therefore
in mastering vinyl records. The phase and amplitude
evolution’s answer to the sun. The ear was the organ
of stereo signals were graphically illustrated. In digital synesthesia the psychophysical parallels
to process sound waves, also a very small spectrum. Our natural sensory organs are very limited. They do
between acoustic and visual stimuli happen at the
not perceive e.g., the so-called radio waves, not to
purely physical, i.e., mathematical level, and are
mention the cosmic microwave background radiation,
arrived at by means of a computer-based scheme. As
the oldest light in the universe. Similarly, there is in
an example we can consider the work by Ryoji Ikeda.
the analog world a correlation between instrument
In his series test pattern (since 2008) he uses a system
and sound, e.g., a piano makes piano sound, and
that converts any type of data (text, sounds, photo-
between object and sound, e.g., each car creates a
graphs, and films) into barcode patterns and binary
typical sound.
patterns of zeros and ones in real time. Ikeda presents intense, flickering, black-and-white imagery that
In the digital world the tie between sensory organ and sensory stimuli, between instruments and sound,
93
objects and sound, is completely broken and inter-
produce any sound. A chair can sound like a piano,
rupted. The source of the sound and the sound itself
a piano can sound like a violin, a violin can sound like
have no one-to-one relationship anymore, because in
a drum, and a glass of wine can sound like wind. All
the digital age the computer turns the sound into data
these acoustic phenomena can also produce visual
and the computer decides which sensory modality
phenomena in real time. The visual phenomena can
is chosen. The computer can turn sound data into
be fed back to the sound sources and change the
light and visual data into sound. The computer has
sound. These sounds can be fed back to the visual
become the universal organ. A piano can deliver
sources and change the visuals. We can create cycles
typical piano sounds, the computer can process them
and circuits of sounds and light, of music and colors
and turn them into violin sounds and additionally
in infinite variations, which only rely on Relay and
create visual patterns. Digital synesthesia is locally
Switching Circuits (Claude E. Shannon, 1938), that is
bound neither to the brain nor to any instrument.
on electromagnetic waves.
The universal brain or instrument is the computer.
Digital synesthesia implies that music is a variable,
The human sensory organs are replaced by technical
not even bound 1:1 to an instrument or object. Like
man-made sensors. We can place a sensor on any
noisy radar data, acoustic data becomes a matter of statistics. Noise is part of any signal and what is noise and what is music always remains uncertain.
perception (eyes or ears) is decided by the computer.
Noise and Randomness, Information and Incomplete-
The computer can modulate the sound, can store
ness, Computability, and Undecidability form the
it, and can rhythmize it. Therefore any object can
parameters in the age of digital synesthesia.
1 Isaac Newton, Opticks: Or, a Treatise of the Reflexions, Refractions, Inflexions and Colours of Light, London, 1704, p. 1.
3 Isaac Newton, Opticks: Or, a Treatise of the Reflexions, Refractions, Inflexions and Colours of Light, London, 1704, p. 109.
2 Roger Bacon, Opus majus [1266], transl. by Robert Belle Burke. Vol. II, New York: Russell & Russell 1962, part 6, ch. 1, p. 583.
4 Ibid., p. 111.
6 Translated into English by Herbert Sidney Langfeld: Elements of Psychophysics [1860], 1912. First published in: Benjamin Rand (Ed.), The Classical Psychologists. Boston: Houghton Mifflin, pp. 562–572.
5 Translated into English by Alexander J. Ellis: On the Sensations of Tone as a Physiological Basis for the Theory of Music, London: Longmans, Green, and Co., 1875.
7 James Clerk Maxwell, “A Dynamical Theory of the Electromagnetic Field,” in: Philosophical Transactions of the Royal Society of London, 155, 1865, submitted in 1864, pp. 459–512.
Peter Weibel
object and any object can produce sound – but what happens to the sound before it arrives at our organs of
THEORIES ON DIGITAL SYNESTHESIA
94
9 Isaac Newton: “Newton to Oldenburg. 6 February 1671/2. From the Philosophical Transactions, 6 (1671/2), 3075–87”, in: H.W. Turnbull (ed.): The Correspondence of Isaac Newton. Vol. I: 1661–1675. Cambridge: University Press 1959, pp. 92–107, here p. 94.
Digital Synesthesia
8 See e.g., p. 36 in: Ryoji Ikeda and Carsten Nicolai, cyclo id. Berlin: Gestalten 2011.
Fig. 1
Fig. 1 Peter Weibel: The Origin of Noise – The Noise of the Origin, 3D-noise concert, world premiere at Donaufestival Krems Brûlée, 25.04.2013. ZKM | Karlsruhe 2013 ©: ZKM | Karlsruhe Photo: Moritz Büchner
Fig. 2 Peter Weibel: The Origin of Noise – The Noise of the Origin, 3D-performance opera at BEYOND Festival, Karlsruhe, 02.10.2013. ZKM | Karlsruhe 2015 ©: ZKM | Karlsruhe Still image: Christina Zartmann
Fig. 2
95
Peter Weibel
Fig. 3
Digital Synesthesia
Fig. 4
Fig. 5
THEORIES ON DIGITAL SYNESTHESIA
96
Fig. 3 Ryoji Ikeda: the planck universe [macro], 2015 Audio-visual wall projection, 3 video projectors, computer, loudspeaker (Exhibition view “Ryoji Ikeda: micro | macro” ZKM | Karlsruhe 2015) © ZKM | Karlsruhe Photo: Martin Wagenhan
Fig. 4 Test protocol by Heinrich Hertz, showing the measuring results of the development of electromagnetic waves. (General Photograph Collection of the Karlsruhe Institute of Technology (KIT), KIT-Archive)
Fig. 6
Fig. 5 Isaac Newton’s Experimentum Crucis from: “Newton to Oldenburg. 6 February 1671 /2. From the Philosophical Transactions, 6 (1671 /2), 3075 – 87”, in: H.W. Turnbull (ed.): The Correspondence of Isaac Newton. Vol. I: 1661 – 1675. Cambridge: University Press 1959, 92–107, here 106a
Fig. 6 The fields of a dipole antenna, from: Heinrich Hertz: “The Forces of Electric Oscillations, Treated According to Maxwell’s Theory” [1889], in: Id.: Electric Waves. Being Researches on the Propagation of Electric Action with Finite Velocity through Space. Translated by D.E. Jones, London: Macmillan, 1893, chapter 9: 137–159, here: 144 –145
Fig. 7 Experimental set-up: Transmitter and two coupled circuits. Photograph taken in Hertz’s laboratory of his Physics Institute at Karlsruhe, around 1887. (General Photograph Collection of the Karlsruhe Institute of Technology (KIT), KIT-Archive)
Fig. 7
97
This book chapter combines ten theor which have been written by neuro- an media and art theorists. The articles re between media aesthetics, synesthetic modalities, theories of embodiment, d issues in the context of DIGITAL SYNE of the international DIGITAL SYNESTH University of Applied Arts Vienna on A Peter Weibel
Fig. 8
Fig. 9 Digital Synesthesia
Fig. 8 Peter Weibel: The Origin of Noise – The Noise of the Origin, 3D-performance opera at BEYOND Festival, Karlsruhe, 02.10.2013. ZKM | Karlsruhe 2013 ©: ZKM | Karlsruhe Photo: ONUK
Fig. 9 Peter Weibel: Data Music, 2016 (Exhibition view AIL Vienna) Photo: Birgit and Peter Kainz
THEORIES ON DIGITAL SYNESTHESIA
98
retical texts on DIGITAL SYNESTHESIA nd cognitive scientists as well as by efer to topics regarding the relationship c perception, brain activities, sensory digital technologies, and philosophical STHESIA. They are also the proceedings HESIA symposium which was held at the April 4 and 5, 2016.
99
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA Part 2
This book chapter introduces 15 DIGIT been produced by 17 renowned media arts-based research process. The digit mental set-ups for the artistic research Texts, sketches and pictures provide in of the works in the research context o
THEORIES
100
THEORIES 102 – 105 Anke Eckardt
VERTICAL 2 106 – 109 Karl Heinz Jeron
Space Time 110 – 113 kondition pluriel
At Play 114 – 117 kondition pluriel
TAL SYNESTHESIA artworks which Diver have 118 – 123 Karen Lancel / Hermen Maat a artists in the course of a three-year E.E.G. KISS 124 – 125 A lan Kwan tal artworks have to be seen as experiThe Flying Umbrella Project h of cross- and multi-modal 126 perception. – 131 Marcello Mercado Bestiary for the Minds of the nformation about the artistic concepts 21 Century: Genomic Opera of DIGITAL SYNESTHESIA. 132 – 135 U lla Rauter st
Sound Calligraphy 136 – 139 R uth Schnell
facades 140 – 143 R uth Schnell
MotU #4–#6 144 – 147 R uth Schnell
Topography of Movement 148 – 149 J effrey Shaw/Sarah Kenderdine
IN_SIDE VIEW 150 – 153 David Strang / Vincent Van Uffelen
Transmission+Interference 154 – 157 Tamiko Thiel / Christoph Reiserer
I am Sound 158 – 162 Peter Weibel
Data Music
101
VERTICAL 2 Anke Eckardt
Anke Eckardt
Artistic media: Installation Primary sensory modalities: Kinesthetics (Balance, Visceroception), Audio, Vision 2016
VERTICAL 2
A
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
102
“How real is
B
A VERTICAL 2 (Sketch, Multisensory Installation by Anke Eckardt)
VERTICAL 2 is a continuation of a series of artworks by Anke Eckardt B VERTICAL 1: ! (Multisensory Installation by Anke Eckardt) A vertical line of six loudspeakers plays a downward sliding tone followed by a heavy sub-bass punch on the ground. The punch seems to trigger an “eruption” in a tank, which is filled with black liquid. The liquid erupts in the form of a round wave, which disintegrates and splashes in all directions. Because of the temporal and spatial conjunction of the two events, the “falling sound” and its “visual impact” right after, we interpret the scene as cause and effect. Synesthetic perception enables the physical experience of an intermodal illusion, realised by means of audio spatialization with multichannel loudspeakers and synchronized pneumatic technology. Photo: Holger Kist
on elements of architecture called SONIC SPACES: VERTICAL 2 (2016), GROUND (2014), WALL (BETWEEN | YOU | AND | ME 2012), VERTICAL 1 (“!” 2010). In this series, the artist explores the ambivalence of sensation. In VERTICAL 2 confusion, disinformation and communication are provoked by a small white cuboid standing in a large exhibition space. It can be entered by one person at a time. Contradictory, even paradoxical spatial information is produced via visual and audible stimuli. Low and small from the outside, the space gains unexpected height and thereby largeness when entered. The absence of consistency between the inner and outer perspective is furthermore deepened by another aspect to be experienced inside. The cuboid not only seems to open up vertically, but also downward below the feet. German philosopher Arthur Schopenhauer broached the issue of architecture as a complex staging system and cultural scientist Hartmut Böhme wrote about spatial atmospheres; not only the arrangement of solid construction material, but also acoustic perspectives and the designed behavior of light play one’s part at this juncture. Famous examples are e.g., St. Paul’s Cathedral in London (by architect Sir Christopher Wren,
103
Anke Eckardt
s real?”
VERTICAL 2
C
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
104
D
consecrated in 1708), which contains a Whispering Gallery, and Swiss C WALL: BETWEEN | YOU | AND | ME (Multisensory Installation by Anke Eckardt) BETWEEN | YOU | AND | ME is a wall of sound and light people can walk through. Four projectors and two lazers are used for creating precisely arranged beams of light. They form a visible frame that is filled with sound. The sonic architecture within the wall, textures of breaking glass, is realised by six extremely vectored hypersonic loudspeakers – a row of overlapping sonic beams and acoustic absorbers nearby.
D GROUND (Multisensory Installation by Anke Eckardt) The ground we stand on is in motion. 96 elements of concrete are moved by immense mechanical forces – motors, air bags, a compressor. The subtle motion can be heard, seen and felt – visitors might experience the loss of their visual reference points, rough sounds are produced just mechanically through friction between the concrete elements, there is an afterglow of a moving ground in the visitors physical memory after leaving the installation.
architect Peter Zumthor’s buildings, with regard to the staging of light or the acoustics of the skyscraper I.M. Pei, Bank of China, built in Hong Kong in 1990. The facade includes a large waterfall dividing into several smaller ones. Not only is surrounding traffic masked, in Chinese philosophy water is also considered to be a medium of immortality, now part of the spatial atmosphere of the Bank. Anke Eckardt precisely stages light and sound and their perceptional interplay into her installations. Inside the small cuboid of VERTICAL 2, the space seems to become an open vertical tunnel. Neuronal information about direction is given via reflected light, information about size via acoustic impulse responses. It might catch the visitor’s eye that the constructional elements of the cuboid are visible from the outside in a reductionist manner. Its building style lies open; there seem to be no secrets. All the more visitors are left alone with the ambivalence of their sensation. The realness of the outside world is considered to be independent of human observations. In this regard VERTICAL 2 appears to be an intentional aesthetic object that questions realness and enables visitors to synthesize virtuality. “In the way that swells of sensory information cause a constant flow of fluid modulations in the human body, Anke Eckardt’s projects function at the interstices of subtle (yet still acute) influence. Precisely interwoven she is using digital media like sound / ultrasound / infrasound and light as artistic material. Her pieces work to pick at the seams of sensorial determinism and reveal the slippage between modes of vibratory affect. In short, her installational projects become exemplars for a new kind of liquid agency; a future-facing state whose shores are lapped by the recent surge in augmented perception.” (Toby Heys)
105
Space Time Karl Heinz Jeron
Karl Heinz Jeron
Artistic media: Radio controlled clocks, custom electronics Primary sensory modalities: Kinesthetics, Chronoception, Audio 2016
Space Time
A
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
106
“I hear the buzz of
In general my artistic work is about reflection and perception. Space Time comprises the competition between two languages mediating complexity. One is the language of science. The other is the language of art, which has to understand science but doesn’t want to become it. The idea of what good science is or should be, the images of science and scientificity, are difficult to grasp but no less significant for it. With my project Space Time, A Studio shot development of 12 radio clock sound objects for Space Time installation
I am interested in the clock drift phenomena. Clock drift refers to several related phenomena where a clock does not run at the exact right speed compared to another clock. That is, after some time the clock “drifts apart” or gradually desynchronizes from the other clock. Everyday clocks such as wristwatches have finite precision. Eventually they require correction to remain accurate. The rate of drift depends on the clock’s quality, sometimes the stability of the power source, the ambient temperature, and other subtle environmental variables. Thus the same clock can have different drift rates at different occasions. Atomic clocks are very precise and have nearly no clock drift. Even the Earth’s rotation rate has more drift and variation in drift than an atomic clock. The principle behind the atomic clock has enabled scientists to define time in terms of a fundamental physical constant, the rate at which a type of particle oscillates. Measuring these events discretely (without skipping an event) is what gives an atomic clock nearly zero drift. Time signal stations synchronize their clocks to coordinated universal time (UTC), the international standard for timekeeping. No clock keeps coordinated universal time exactly because coordinated universal time is an average time, calculated with data collected from hundreds of atomic clocks located around the world. The accumulated time error of an analog radio-controlled clock will be measured between synchronizations. A sensor will be used that records the beat-rate of the watch’s pulses. Comprising several radio-controlled clocks, the Space Time installation will make this inaccuracy audible with clocks, electronic oscillators and speakers. A radio-controlled clock will be most accurate immediately after a successful synchronization and will become less accurate from that point forward until the next synchronization. To demonstrate this, the accumulated time error of an analog radio-controlled watch was measured between synchronizations. A sensor will be used that could record the beat rate of the watch’s pulses. Synchronization accuracy is limited by transmission system delay, path delay and synchronization errors.
107
Karl Heinz Jeron
of the clock drift.”
B
Space Time
C
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
108
Once the time synchronization signal leaves the transmitter and enters free space, it travels at the speed of light to the radio-controlled clock, approximately 0.000 003 336 seconds per kilometer. If you know the location of your radio-controlled clock and the location of the time signal station, you can calculate the distance between the transmitter and the receiver, and estimate the length of this delay. For example, a 3000 kilometer path would delay the time signal by about 10 milliseconds (0.01 s). This assumes, of course, that the signal travels along the ground and covers the shortest possible distance between the station and the radiocontrolled clock. With the low frequencies used by the time signal stations, a groundwave path can be assumed for short distances, perhaps up to 1500 km, but at longer distances the signal might bounce off the ionosphere (skywave) and take slightly longer to arrive. Even so, it is B Loudspeaker, radio clocks, oscillator control for sonification of time synchronizing deviation
safe to assume that the path delay will be less than 20 milliseconds (0.02 s) because the signal will rarely be usable at distances of more than 5000 km, and any additional delays introduced by skywave will be relatively small. There will be a measurable inaccuracy which will be
C Sensor, radio clock, DCF receiver (Installation 2 of 12 radio clock sound objects)
different with different brands and types of radio-controlled clocks. Space Time uses sonification to make time tangible. Sonification is the data-dependent generation of sound, if the transformation is systematic, objective and reproducible, so that it can be used as scientific method. A distinction between data and information is irrelevant with regard to the definition: information like, for instance, a message can always be represented numerically and thus be understood as data. Sonification refers to the technique and the process; the algorithm. Sonifications of the measured sensor data may be heard as music. According to the measured time inaccuracy, the sound, the color of the sound, and the volume of the sound are varied. The time inaccuracy adds a certain chance aspect to Space Time. Chance is the creative spark which excites artistic imagination and causes non-predictable but feasible material results. Thus, a sort of improvised piece of music is created from the simplest electronic parts. The sound installation refers to the concept of aleatory music, based on aspects of chance and improvisation as used by John Cage, Pierre Boulez, and Iannis Xenakis. Space Time depends on Charles Sanders Peirce’s notion of Tychism; rather than postulating the importance of determinism in cosmology and in socio-economic Darwinism.
109
At Play kondition pluriel (Martin Kusch / Marie-Claude Poulin)
kondition pluriel (Martin Kusch / Marie-Claude Poulin)
Artistic media: Responsive audio-visual installation with computer-tracking system Primary sensory modalities: Spatioception (Proprioception, Touch, Balance, Audio, Vision) 2016
At Play
A
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
110
“At the heart of our work is t the performativity existing in
B
At Play – a moment in the life of objects Might the empowerment of objects contribute to questioning our relationship to them, as well as the very meaning of their production? (David Bihanic)1 1 Empowering Users through Design: Interdisciplinary Studies and Combined ..., Ed. David Bihanic, 2015, Springer 2015
Every environment, its components and the relationships between them is enigmatic; a mystery to be penetrated, a space to comprehend. How should a place be read? Which decisions and actions should we carry out, and which relationships should we have? At Play is a responsive audio-visual environment that invites visitors to make stacking combinations with plastic storage bins to build the architectural landscape of the installation. Miniature versions of containers
111
kondition pluriel (Martin Kusch / Marie-Claude Poulin)
s the performativity of bodies, beings, machines and systems, but also in inanimate entities such as places and objects…”
At Play
C
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
112
that carry goods to world markets, these everyday objects, are playfully moved around by visitors to the exhibition like pieces of a puzzle. Resonance chambers, light sources and dynamic projection surfaces all at the same time, the boxes are integrated into a pattern recognition A, B, C “Enjeux” performance at Theatre La Chapelle, Montreal Photo: Bruno Colpron, © kondition pluriel, 2015
system that matches each sculptural composition with a particular visual and sonic atmosphere. The projected images and sound ambiances embed the visitors in fictional worlds in which they become the actors. Playing with building blocks that appear a priori analog, their actions reveal atomized sections of a world undergoing a perpetual mutation, calling upon their spatioception (perception of space). The grey material of the objects, fused with the lighting of the projected 3D imagery, comes to life through multiple trompe-l’œil nuances. The acoustic universe takes its source from the immediate environment of the installation, and both image and sound undergo a series of transformations in real time that imbues the grey boxes with a mysterious expressive power that has political undertones. The visitors act, but also tele-act; navigating in real space and in virtual space, they are guided by their senses of touch, balance, hearing, proprioception and sight. These usual integration mechanisms of their perception are altered by the afflux of multimodal and heterogeneous sensory information, which is not limited to the experience of the real. Instead, the experience oscillates between real and virtual, while the manipulated objects act as limits and physical constraints, but also as tools and partners of the imaginary. The sensorial displacement gener-
Produced by: kondition pluriel in coproduction with the Digital Synesthesia Group With the support of: Conseil des arts et des lettres du Québec (CALQ) Conseil des arts du Canada (CAC) Concept and idea: Martin Kusch and Marie-Claude Poulin Artistic direction: Martin Kusch and Marie-Claude Poulin Interactive design and media content: Martin Kusch and Johannes Hucek Visual programming and technical setup: Johannes Hucek Sound: Alexandre St-Onge
ated by the experience of the installation, a doubling of personality, does not allow the visitors’ body image (the corporeal schema) to stay intact: rather, it becomes dislocated to reincarnate itself in another form, in a synesthetic autopoiesis. The “construction blocks” of At Play, symbolic objects of both childhood and the adult world, evoke our enactment of social roles through our functions, or simply by the way we live, in a globalized world waiting to be reinvented. By proposing an inversion of perspective – an attempt to see the world from the point of view of objects, At Play brings into question our existential dilemma and our relationship with things and places. The boxes ultimately become information units, the visitors potential catalysts of change, and the installation a transformation platform.
113
Diver — a cyber-kinesthetic installation kondition pluriel (Martin Kusch / Marie-Claude Poulin)
Diver — a cyber-kinesthetic installation
kondition pluriel (Martin Kusch / Marie-Claude Poulin)
Artistic media: Various options (360˚ immersive full-dome projection installation; dynamic multi-screen installation; giant urban screen application) Primary sensory modalities: Kinesthetics, Vision, Audio 2016
A
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
114
“By crisscrossing assignations between media-da fabric of relationships in which all the elements
It is difficult to understand how the sense of corporeal movement, or kinesthesia, is commonly excluded from the list of the traditional five senses, even though it is essential to our survival. A plausible reason for its omission is its reliance on multiple receptors located deep within the body (muscles, joints, inner ear). Also, unlike the case for the other senses, corporeal movement is not generally perceived consciously. (cf. A. Berthoz) Diver is a transdisciplinary interactive media installation that gives eyes to multiple viewpoints captured from the body of a competition springboard diver. The images, sound and biometric data – like velocity, kinesthetic inertia and gravity – as well as camera-tracking data originating from the diver’s body, are the source-material for the installation. The piece is adaptable to three installation formats: a 360˚ immersive fulldome projection installation; a dynamic multi-screen installation in a gallery setting; and a giant urban screen application. By moving about or by using their smart-phones, visitors and passers-by modify simultaneously transmitted video and audio recordings from the various viewpoints, transforming and recombining the media material, drastically slowing it down or speeding it up. The orchestration of these mobile and dynamic viewpoints offers a myriad of possibilities for composition; in collaboration A kondition pluriel: Diver (Sketch 1), 2016 © kondition pluriel
with other viewers, the participants become the co-authors of an augmented visuo-audio-kinesthetic work. Regarding her video-installation machine-vision, Steina Vasulka said: “When a human being operates the camera, the assumption is that the camera is an extension of the eye. You move the camera the way you move the head and the body” (1974). In her work she mounted motorized mirrored spheres in front of camera lenses, simulating all possible camera movements, and spoke of “freeing the human eye from being the central
115
point of the universe.” Today we have reached a cultural state where technological developments lead to a hyper-media reality and our perception expands beyond the micro- and macroscopic level. The perfect simulation takes over the imperfect reality and we live in a multisensorial state of existence. With eight cameras installed on the surface of a diver’s athletic body, his limbs, joints and skin transform into a myriad of eyes. This policed
kondition pluriel (Martin Kusch / Marie-Claude Poulin)
-data and control-data, we weave an intricate nts in place resonate and answer in organic ways.”
body, a metaphor for the struggle for control of the physical body, becomes “the center of the universe.” The movements of this perfectly controlled and hyper-conditioned body become the motor for delivering an insight into that which we are not able to see or perceive. Diver
kinesthesia into an uncommon union, in which these senses are the extension, or the mirror of each other. This project deals with systems of control and the human body as a metaphor of that control, and it questions our connection to authorship and the autonomy of humans and machines. Data captured, analyzed and choreographed by machines are part of this process. The project reflects on the tension between representation and performance, between the organic and the mechanical and between centralized and distributed control. In Diver, the viewer is invited to explore the materiality of media in real time, and the installation offers a participative multi-sensorial experience of a hybrid reality that incorporates the performative and event-like
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
116
Diver — a cyber-kinesthetic installation
brings into being a hyper-body with multiple body-eyes that become the extensions of the viewer’s eyes. Sight and hearing are coupled with
B
gestures of an athletic body. A blend of human and technical presence, this body has no “real” existence or single identity, no personal history and possibly no real future in our society of surveillance and spectacle. B kondition pluriel: Diver (Sketch 2), 2016 © kondition pluriel
However, the installation apparatus dismantles this anonymous, normative corporeal condition: instead of simply stimulating the observer’s vision in isolation, the extreme slowness of the source images and the rhythm and sequenced timing of the media elements create new narratives, potentially touching the senses and deep-seated memories of the viewer. Paradoxically and unexpectedly, the installation conveys a decolonized kinesthesia close to Walter Benjamin’s flâneur: a trembling, floating, vacillating body; an environment-architecture oscillating between background
Produced by: kondition pluriel in coproduction with the Digital Synesthesia Group With the support of: Conseil des Arts et des Lettres du Québec (CALQ) Conseil des Arts du Canada (CAC) Concept and idea: Martin Kusch Artistic direction: Martin Kusch and Marie-Claude Poulin Interactive design and media content: Martin Kusch and Johannes Hucek Visual programming and media setup: Johannes Hucek and Martin Kusch Sound: Alexandre St-Onge and Martin Kusch Diver: N.N.
and foreground; the experience of the dizziness of falling; images and sounds suspended in time and space. Diver offers an exploration of sensory relations within an elastic notion of time, of perception from the inside rather than the limited perception of the physical exterior of things. In Diver, positioning and tracking technology separates the components of the action, then recombines them in various layers to create relationships between the media-created space, the recorded space, and the real space. The generic athletic body is made accessible to the nonathletic bodies of the viewers via their physical involvement. The repeated images of the dive transform into an inextricable post-human machine performance. What we see is a hyper-networked-body, encased in environments both immediate and distant; a social and autopoietic entity revealing cyber-kinesthesia, the fruit of an integrative digital visuo-audiokinesthetic synesthesia that allows us to experience other possibles.
117
E.E.G. KISS Karen Lancel / Hermen Maat
Karen Lancel / Hermen Maat
Artistic media: Performance, installation Primary sensory modalities: Taste, Touch, Vision, Audio 2016
E.E.G. KISS
A
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
118
“Can I kiss y
Can I kiss you online? Can we measure a kiss and what kissers feel together? Can we transfer a kiss and its intimacy online? Do we want to save our private kisses in a transparent database – to be used by others? In E.E.G. KISS we explore how a kiss can be translated into data. In a poetic, electric environment for kissing and measuring, for synchronizing and merging, we research a new sensory synthesis for networked kissing. A digital synesthetic kissing ritual In E.E.G. KISS we deconstruct and turn around control technologies and sensory perception, to facilitate a new intimate kissing experience. Worldwide, in performances and live kissing experiments with E.E.G. sensors, we invite visitors to kiss. While kissing, their brainwaves are measured and visible in real time as streaming E.E.G. data. These data are projected around the kissers as an immersive environment, a “Kissing Cloud”, creating an aesthetic and spatially immersive experience. All witnesses of the A While kissing, the kisser’s brainwaves are measured and visible in real time as streaming E.E.G. data. Photo: Lancel /Maat 2014; Eye Film Institute Amsterdam / Discovery Festival 2014
kiss are invited to be both voyeurs of the kiss and scanners of E.E.G. data. The E.E.G. data are translated to a music score, based on an algorithm, which is further translated to a soundscape. The soundscape is generated by the kissers’ streaming E.E.G. data. In this kissing ritual both kissers and witnesses share a trans-sensory experience of “kissing”, and “E.E.G. data”. Together they feel, see, hear and mirror a communal, networked kiss. The soundscape has been developed with Tijs Ham at STEIM Amsterdam. Intimacy and Big Data: critical reflections A kiss is an obvious case of emotional interaction and synchronizing, spatial nearness, touch, sight, fluids, smell. When our kisses are being quantified, on what data-visualization design do we validate this interaction? Who is responsible for and who will benefit from our quantified kisses? And how can we trust our own kiss data? In E.E.G. KISS we invite
119
Karen Lancel / Hermen Maat
s you online?”
E.E.G. KISS
B
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
120
C
participants to give meaning to the abstract, even mystifying E.E.G. data-visualization, based on the embodied memory of the kiss as a form of “intimate co-creation”. and ‘co-attribution’. Each unique E.E.G. KISS data stream is saved in a database and can be printed showing “a portrait of a kiss”. Share your private kiss Each E.E.G. KISS soundscape is saved in a public database on the artists’ website to be downloaded and shared. Art, Science & Technology E.E.G. KISS emerged from further artistic research that we conducted at Delft University of Technology in the context of Lancel’s PhD trajectory (Promoters: Prof. Dr. Frances Brazier, Dr. Caroline Nevejan) and is further developed with innovative sensor technologies, in cooperation with neuroscientists and partners in art, science and technology. To inspire B Frascati theaters Amsterdam “End of Privacy festival 2016” Photo: Anna Van Kooij
and rethink sustainable socio-technical ecological systems, we deconstruct automated control technologies (such as surveillance and social media, biometric technologies and brain-computer interfaces) to inspire sensitive relations based on intimacy, digital synesthesia, tacit knowledge,
C EEG KISS participatory, shared ritual for a trans-sensory experience of “kissing”, “heartbeats” and “EEG /E CG data”. Photo: Roy Te Lintelo 2015; (At Gogbot Festival 2015 “Internet of Things / Theories of Future Visions”)
bio-synchronization and aesthetic perception. In this way, we explore digital synesthetic “trust systems” based on holistic experience. Our aim is to create Meeting Places and Meeting Rituals for sensitive networked public space, with shared response-ability for the power of reciprocity and synchronizing, through touching, breathing, kissing, dancing, sharing space and presence.
121
Karen Lancel / Hermen Maat D
Notes on the E.E.G. KISS Project by Lancel/Maat. “When I kiss you, I can taste your soul.” (Terri Guillemets) One starting point for discussing Lancel /Maat’s E.E.G. KISS project could much the cultural and regional differences (as brilliantly documented in Lev Manovich’s Selfie City installation) but the technological homogeneity of these selfie moments. The pressures on the subject-as-user to share the compulsive exhibition of the present with others results in
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
122
E.E.G. KISS
be its contrast to the selfie. What is striking about the selfie cult is not so
a perverse update of Vilem Flusser’s apparatus concept in photography theory. What is expressed in the compulsive selfie is not just the template culture of the social media platforms and technical specifications of smartphone lenses and the (default) settings of their software, but also the victory of logistics, with its massive online services, broadband and D Photo: Lancel / Maat 2015; At Brakke Grond / Frascati Theatre Amsterdam (Yami-ICHI and Privacy Festival)
data-centers that all “eat into” the image in an unconscious manner. Instead of the ego presence of the selfie, E.E.G. KISS presents us with a shift towards an inner experience with the other. Lancel and Maat’s techno-performance of colliding wetware captures the intimate act of kissing in a time-based fashion. Their autograph of love emphasizes the social aspect: we never kiss alone. This is no auto-erotic kiss but a collective experiment that goes way beyond the self-centered exhibitionism that defines our social media age. The installation opens up a space of free interpretation. Despite the neuroscientific look and feel, users have to be made aware that the produced graphs of their kiss do not contain an eternal truth. The displayed data are imagined intensities; never medical evidence of whatever. The data visualizations merely represent our obsessive techno-culture. In an epic quest to invent new interfaces and tools that will help us to develop the visual language that defines our era, some artists are willing to take risks — not just with their reputation. They ask the question: what’s a quantified kiss? The risk in the case of E.E.G. KISS is an obvious one and goes beyond the danger of being appropriated by “the system”. What if
In collaboration with and the generous support of: Digital Synesthesia Group, University of Applied Arts Vienna; Mondriaan Foundation (Vrije beurs praktijverdieping); STEIM Amsterdam; Baltan Laboratories Eindhoven; Fourtress Eindhoven; Holst Centre Eindhoven (sponsor E.E.G. headsets IMEC); University for Technology, Delft / Participatory Systems Initiative; Discovery Festival, EYE Film Institute Amsterdam; Neuro-engineering lab Tsinghua University Beijing; Social and Affective Touch Research Group TNO (Netherlands Organization for Applied Scientific Research); Artists-in-residency TASML Tsinghua Art and Science Media Laboratory, Tsinghua University Beijing
our kisses are stored in databases against our will and analyzed by algorithms owned by banks that decide on home loan applications? An all-too-passionate kiss can indicate that the wild romance will end up in a dramatic divorce. A flat and boring kiss will transmit the same message. This puts the motive of the “stolen kisses” in another context. All too easily, networked data facilitate the most absurd readings into a situation. Rebel souls will arise and claim that united spirits will never be documented. Why store an act of human intimacy in a machine, when the real issue is our acute loss of memory — of any experience? How can we train our bodies to, again, store our vital data? Instead of administrating the last remaining expressions of us, being human, artists arise against the technological elimination of our instincts. The quantified artist is the problematic figure here, the willing researchers that smoothen out the clean agenda of the virtual sex industries. Instead, let’s investigate together how the fluids, flowing together, can short-circuit our cynical architectures of power. As Billy Holliday said: “A kiss that is never tasted, is forever and ever wasted.” (Geert Lovink)
123
The Flying Umbrella Project Alan Kwan Artistic media: Aerial Robotic Performance Primary sensory modalities: Kinesthetics, Touch, Audio 2016
The Flying Umbrella Project
Alan Kwan
“An erotic synesthetic performa
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
124
ance that takes place in mid-air.”
The Flying Umbrella Project is an outdoor surreal erotic performance that takes place in mid-air. The performance consists of two robotic flying creatures in the form of umbrellas mating in the air. Created with customized flying ma Artist: Alan Kwan Artistic & Engineering Advisor: Bjorn Sparrman Engineering Advisor: Roger Wang Sound Designer: Jose Alejandro Rivera In co-operation with TONSPUR Kunstverein Wien (Artistic director: Georg Weckwerth) / Q 21 (at MuseumsQuartier)
chines, control algorithms, and animatronics, these umbrella robots kiss, embrace, caress, and copulate with each other in the air. At the same time, specific sound frequencies are generated digitally from these body motions and intimate contacts, allowing audiences to not only see but also hear the sensuality and lust of these artificial life organisms. The performance therefore explores the idea of robot pornography, and also investigates the use of flying robots in creating motion-sound crossmodal perceptual experiences for artistic aerial performance.
125
Bestiary for the Minds of the 21st Century: Genomic Opera A data-mining, synesthetic, 3D-printed genetic opera Marcello Mercado
Bestiary for the Minds of the 21 st Century: Genomic Opera
Marcello Mercado
Artistic media: 2-channel video installation, artist’s book, public audio interaction Primary sensory modalities: Kinesthetics, Touch, Vision, Audio 2015
A
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
126
“Digital Synesthesia is a data mining process, trans
We live in a biodiversity crisis: A massive daily loss of biological data has led me to think about a genomic opera that could preserve/archive fragments of thousands of prokaryotic and eukaryotic genomes (like a Noah’s Ark) that can be combined and transformed into musical notations in the form of tactile 3D-printed objects. The aim of the art project is to create 3D-printed objects based on various mixed genomes, as well as to explore the possibilities of 3Dprinting errors (glitches). Fragments of the genetic information of different types (mammals, insects, fungi, bacteria, etc.) are randomly mixed, and by means of algorithms transformed into audio files which are then converted into 3D-printed objects. These environment-friendly 3Dprinting filaments, combined with glitches and incomplete 3D-printed remains, make up the score of the musical notations. I researched the available genome databases like Ensambl, Genome, Wormbase, Flybase, etc. to mix and cut eukaryotic and prokaryotic genomes in random combinations. The nitrogenous bases (as the letters A, T, G, C) are read by a software that transforms text into audio files. These are transformed into 3D objects, which are again printed in plastic with a 3D printer and pasted onto paper as music notations. The generation of AUDIO and VIDEO through TACTILE actions involves audience participation: The audience interacts by rubbing the 3D-printed objects with a piezoelectric microphone connected to a transductor to visualize new sounds. In parallel, each page of the opera is photographed and scanned with sonogram software in order to be converted into sound A ABS-plastic (ABS) and polylactic acid (PLA) (bioplastic) on polymethyl methacrylate (PMMA), copper colored, 2015 Page from the Opera-Artist’s Book: 3D printing errors (glitches), 3D remains and data mining process on genomes from mammals, insects, fungi and bacteria converted into plastic 3D-printed objects (40 cm x 60 cm x 30 cm)
archives.Thus, the final performance of the opera is a combination of: • 3D-plastic-printed genome-data-mining-instruments, performed live by a musician rubbing the 3D printed objects with piezoelectric microphones, • Recorded mixed genome fragments, • Sonogram audio files, and • The score that instructs the performers during the entire duration of the piece throughout the four acts.
127
Bestiary for the Minds of the 21 st Century: Genomic Opera
Marcello Mercado
anslated into the space by means of algorithms.”
B
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
128
On Marcello Mercado’s Genomic® Opera: Bestiary for the Minds of the 21st Century For a long time dualism served as a convenient substitute for thought. We distinguished between artists whose creative process is driven by their minds and those who rely on their gut instinct. Defined in terms of media theory: A software and a hardware approach were interpreted as oppositional and frequently antagonistic positions. This was the 20th century analogy to the body-soul dualism of the European Middle Ages which branched out into different guises in the modern age. The subversive perspective of alchemy, for example, vehemently contested this dualism even in the pre-modern era. The tradition of pre-Socratic philosophy in Greek antiquity supposed that the soul was flesh through and through, where matter was considered nothing more than modified spirit embodied in various forms and conversely spirit was only a subtle formulation of matter. The disciples and sorcerer’s apprentices of alchemy developed this thought further into an experimental practice and an enigmatic Hermetic paradigm. The lapis philosophorum, the philosopher’s stone, which is thrown onto stubborn matter in the last B ABS-plastic (ABS) and polylactic acid (PLA) (bioplastic), gouache and pencil on transparent paper, 2015 Pages from the Opera-Artist’s Book: 3D printing errors (glitches), 3D remains and data mining process on genomes from mammals, insects, fungi and bacteria converted into plastic 3D-printed objects (40 cm x 60 cm x 30 cm)
stage of transmutation to refine and instill it with metaphysical meaning, is both: the powdered matter of transformation and an algorithm in the sense of an order which was distilled on the basis of scientific experiments. Argentinian artist Marcello Mercado is working in such a subversive tradition, exhibiting radical consistency as much as exuberant energy and ingenuity. In a first performance which I saw almost 20 years ago, he confronted the complex electronic and digital machines of a post-processing unit costing millions with his naked body and the urges of the mind imprisoned within it and clamoring to get out. With all the power of his sensory perception and experience he dedicated himself to generating worlds of images and sound and developed with Das Kapital [Capital] a gigantic opus which cannot end, because it has become an obsession and is thus not so
129
D
E
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
Bestiary for the Minds of the 21 st Century: Genomic Opera
Marcello Mercado
C
130
much organized in a linear as a dynamic and chaotic fashion. The visual and acoustic language that M2 invented for his extremely idiosyncratic interpretation of the Kapital as a single political economy of the body is a continuous process of transforming data and relations into sensations and the aggression to be experienced; all of this has proliferated excessively, already yielding more than 40 hours of recordings. Simultaneously to the transformative work of Das Kapital, M2 has over the last ten years experimented with genomic material. His studio has be
come a biological and chemical research lab; he dug outdoors, extracted the lifeblood of plants and animals and started to obsessively break them
C Artist’s Book
down into their individual components and combine them, much like he does with data clusters in his computer and the recordings of his analog sound equipment. Gay film legend Fassbinder and heterosexual Fluxus
D Pencil on transparent paper, 2015 Page from the Opera-Artist’s Book: 3D printing errors (glitches), 3D remains and data mining process on genomes from mammals, insects, fungi and bacteria converted into plastic 3D-printed objects (40 cm x 60 cm x 30 cm)
legend Vostell never came together in reality. M2 mixed their genetic information using extracts distilled from flowers growing on their graves and merged them in drawings which he produced using the extracted genomic material, blurring the boundaries between virtuality and reality. M2’s Bestiary for the Minds of the 21st Century is the culmination of combining various strands of his work over recent decades. Much like in the case of Kapital, which he composed as an oratorio and for which he
E Gouache, acrylic paint and pencil on transparent paper, 2015 Page from the Opera-Artist’s Book: 3D printing errors (glitches), 3D remains and data mining process on genomes from mammals, insects, fungi and bacteria converted into plastic 3D-printed objects (40 cm x 60 cm x 30 cm)
wrote several libretti, he advances his opera experiment as an aesthetic genre in which precision and passion interact in an extraordinary manner. Sound is translated into visual patterns. In contrast to the legendary figures of Ernst Florens Friedrich Chladnis from the 18th century, he renders the temporal variability and audible intervals not merely on canvas, but in 3D-objects. Once transduced into data files, the sounds are printed out as oblong sculptures in bright colors, which on a notation can be experienced as wavelike information clusters in the truest sense of the word. Using piezoelectric microphones as an interface, the opera audience is able to scan the surface of the material. The experimental apparatus in turn converts the sound thus produced into graphical structures, producing an entirely different experience of the original composition for the viewer, as the result of the tactile interaction now visualized in the resulting audio/visual representation. Another interpretative aspect is that with this Genomic Opera M2 seeks to contribute to evolving the diversity of species in our biological world, which he sees as threatened by the contemporary drive to standardization. However, the knowing of this is not a necessary prerequisite for the aesthetic and intellectual enjoyment of the composition. Much like in Das Kapital, his impetus toward an undisciplined heterogeneity becomes a tightrope act, unfolding above the abyss of our reality, oscillating between program and imagination as well as between rule and ingenuity rebelling against censorship. (Siegfried Zielinski)
131
Sound Calligraphy Ulla Rauter
Sound Calligraphy
Ulla Rauter
Artistic media: Performance, drawing, live-sonification Primary sensory modalities: Vision, Audio, Kinesthetics 2015 – 2016
A
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
132
“The translation from one medium into anoth process, which is able to reveal new aspects
The project consists of calligraphic handwriting containing the spectral sound information of spoken words, which can be made audible by techniques of sonification. During a live performance, I draw calligraphic handwriting which is trans-formed into sound by means of a special scanner (camera and computer-software). I work at the interface between sound and fine art, exploring processes of transformation. In earlier works I focused on a special kind of sound visualization, on spectrographs (also called voiceprints): A spectrograph is a 2-dimensional time-frequency analysis of a sound. I am not only fascinated by the aesthetics of these images but also by their function as a medium – like a code, they contain a certain message as well as the information about the original sound. With certain technology it is possible to convert them into sound again and recover their acoustic contents.
In my work Sound Drawings (2010), I began to create hand-drawn spectrograms: I reversed the process of sound-visualization and painted spec-
A Sound Calligraphy, Performance (Sketch)
trographs of a human voice on the wall, which afterwards were made audible through an inverse Fast Fourier transformation.
133
Ulla Rauter
other is like an alchemistic ts of the source material.”
Sound Calligraphy
B
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
134
C
My interest lies in the fact that the combination of manual drawing and digital technology is able to create an artificial voice and the perception of speech. Apart from the poetic performative aspect of drawing voices, human speech and its perception are part of the focus of my interest.
My drawings are in fact only layers of frequencies that imitate a human voice. It is not much more than noise and a cluster of sine waves, and it
B Sound Drawings, 2010
C Sound Calligraphies (Sketches)
is artificial. But at a certain point in time, we can start to recognize words, maybe a sentence; we start to hear human speech. Due to the constructive behavior of the brain, we search for meaningful messages within the noise that we hear.
135
facades Ruth Schnell
Ruth Schnell
Artistic media: Audiovisual animation, immersive Fulldome projection Primary sensory modalities: Vision, Audio, Kinesthetics (Balance, Visceroception) 2016
facades
A
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
136
“Different motion parameters interact. The result is a dramatur
A facades (2015), performed in the context of Liminal Spaces – E /M /D/ L, Satosphère, Montreal Photo: Johnny Ranger
Constructed from arrangements of point clouds, a virtual urban scene is developed in facades. The viewers seem like flâneurs, following streets that are flanked by classical facades. The movement of the viewers is virtual and is generated by the movement of the projected image space. facades is an audiovisual animation, conceived for performance in a Fulldome. Fulldomes are immersive, dome-based projection environments which – as an interface – allow the full and multimodal integration of the viewers in a multi-sensory, potentially interactive experiential space. Ruth Schnell makes use of the properties of this medium, which is comparatively young in the domain of the arts, to explore and challenge traditional concepts of space, reality and perception – a central theme of her art. Ruth Schnell has designed a dramaturgy of disintegration and reconstruction. The seemingly “organic” facades and streetscapes appear elastic; diaphanous rows of houses overlay each other, panoramas may unfold. The viewer can move virtually in a horizontal fashion or into the depth of the configuration. The individual motion of the projected content
Concept and idea: Ruth Schnell Artistic direction: Ruth Schnell Media content and animation: Nikola Tasic Media setup: Martin Kusch, Johannes Hucek Sound: Alexandre St-Onge, Marie-Claude Poulin
simulates both spatial dynamics as well as a first-person perspective. Sound accompanies this moving image. It is based on the recording of the text No Such Agency, published in 1986 by media theorist Friedrich Kittler about the NSA. Whispered fragments of the text are acoustically placed in the projection, seeming to drift past, and coming together in the virtual space.
137
Ruth Schnell
rgy of disintegration, an experience of instability.”
facades
B
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
138
C
The image content remains open toward the top and bottom; without the reference points of “sky” and “ground”, spatial delineation is undermined. The presented immersive experience is also an experience of instability. The imagery of facades has its starting point in 3D models of city buildings, developed using a photogrammetric method. Here, through several calculations and various software programs, 3D information B facades (2015), performed in the context of Liminal Spaces – E /M /D/ L, Satosphère, Montreal Photo: Johnny Ranger
on an object is derived from only two-dimensional images. The 3D models, generated from point clouds and combined with a 3D graphics program to form streetscapes, allow changes in perspective and various simulated movements. facades was originally developed for dome projection in Montreal’s Satosphère, a Fulldome 18 meters in diameter with a 360-degree
C facades (2015), animation still Photo: Nikola Tasic
expansion of the image on the horizon and 210 degrees at the zenith. (Patricia Köstring)
139
MotU #4 – #6 Ruth Schnell
Ruth Schnell
Artistic media: Light objects Primary sensory modalities: Vision, Kinesthetics 2011/2016
MotU #4– #6
A
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
140
“The habitual process of visio
B
The term “to see” is often used in common speech synonymously with “to understand” or “to grasp”. The phrase “If I am seeing this correctly” is used to signify understanding in consensus with others, while those who “don’t believe their eyes” experience something unexpected and untoward. But in general we do believe our eyes; so much so that we consider the information our visual senses present us with to be an accurate image of the A MotU #3 (2011), solo exhibition, Galerie Grita Insam, Vienna Photo: Birgit and Peter Kainz
present moment, to be reality.
B MotU #2 (2011), solo exhibition, Galerie Grita Insam, Vienna Photo: Birgit and Peter Kainz
emerge from three flickering LED beams and float hologram-like in front
The work MotU #4 – #6 from the series Mirrors of the Unseen challenges this supposed credibility of the visible. The habitual process of seeing is infiltrated by words and icons made of light; they seem to of the surrounding architecture. Nothing can get in the way of these translucent (written) images. They are present, and yet at the same time do not exist: only individual perception makes them accessible to the viewer. Ruth Schnell started developing so-called “light sticks” at the beginning of the 1990s. Since then, the works created using these media have been technically advanced; they befuddle our idea of perception.
141
Ruth Schnell
sion is infiltrated by words and icons made of light.”
MotU #4– #6
C
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
142
The light stick installations make use of the inertia of the human eye: The photoreceptors of the retina cannot resolve separate frames at a rate of more than about 20 frames per second. This is the reason why we see film as film, why the annular motion of a flashlight in the dark appears to describe a closed circle of light. For the display on the light sticks, words, icons and also schematic photographic images are broken down into points that are successively shown on the vertical columns of the beam. The characters thus converted into a temporal sequence of highfrequency pulses then become readable so long as the viewer forgoes habituated reception patterns practiced through the act of reading. What this necessitates is a free-floating gaze, an observation in passing. While moving, the viewer takes in an afterimage, thus seeing several past situations as a continuous present event. The performative act of moving alone creates this “present” and thus also the work as a collaborative process. In MotU, the viewer is therefore particularly involved in the creation of the work. The icons and images on the sticks alternate at a certain frequency. The afterimage is distorted in direct relation to the individual’s perspective. There is no collective viewing of the work; what is seen is an individual experience rather than something tangible or verifiable. Site- and situation-specific works and series of works with light sticks C All Targets Defined (2006), solo exhibition, Galerie Grita Insam, Vienna Photo: Birgit and Peter Kainz
are created for very diverse contexts, particularly responding also to the viewers’ expected movement patterns. Large-scale light stick installations have been conceived for public space with works such as Floating Signs (Bregenz harbor) or the interactive installation laut und leise (Dornbirn indoor pool). Several small beams, arranged in relation to students’ and employees’ rhythms of movement, are being used in the commissioned art projects for HTL Bregenz (polytechnic) or the EnBW’s headquarters in Stuttgart.
Concept and idea: Ruth Schnell Artistic direction: Ruth Schnell Media content: Ruth Schnell, Lea Schnell, Patricia Köstring Programming and media setup: Stefan Istvanits, Alexander Pausch
With regards to content, the works weave a fabric of thematic representation. The words and icons projected onto the sticks for MotU #4 – #6 include terms from the field of synesthesia which are associatively condensed through sequencing and rhythmization. (Patricia Köstring / Ruth Schnell)
143
Topography of Movement Ruth Schnell
Topography of Movement
Ruth Schnell
Artistic media: Dynamic projection Primary sensory modalities: Vision, Audio, Kinesthetics 2016
A
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
144
“The dynamic projection does not interpret the architectural space as background
A Gegen die Zeit (2008), Tiefenrausch exhibition, a project by Offenes Kulturhaus and Linz09, Linz Photo: Otto Saxinger
A hand moves, it explores its surroundings, traces the surface on which it lies, it acts. Based on this actually simple cinematographic process, in Topography of Movement conventional perspectives of the two-dimensional (the motion picture) and the three-dimensional (the real environment) are altered. The relationship between image projection and screen changes, the space is reconstructed by the cinematic sequence. For her dynamic image projections, Ruth Schnell works mostly with computerized projection mirrors. The image content itself is cropped and projected without a background. It moves about in manifold ways: In Topography of Movement projected hands perform an activity. The
Concept and idea: Ruth Schnell Assistance: Patricia Köstring Artistic direction: Ruth Schnell Media Content: Ruth Schnell, Patricia Köstring Postproduction: Alexandre St-Onge Camera: Azalea Ortega, Nikita Zhukovskiy Performer: Marie-Claude Poulin, Nikita Zhukovskiy
sequence in turn is dynamized, accelerated or counteracted via the movements of the projection mirror, depending on how the two movement patterns come together. Digital image production and a spatial reference system intertwine via these overlapping movements. The image projection does not interpret the architectural space as a background or screen, but as an element of a new image formation; the projection itself is conceived not as a frame, but as an independent movement of unfettered images that can theoretically occupy any part of the room. So the real space with the help of the moving projection is newly modeled because, depending on the projection angle and surface structure, distortions and fragmentations of the image content arise in the contrast between projection and architecture. The multiple projection (a right and a left hand), the scaling of the projected scene, as well as the implied tilt of the camera angle from top view to a side view override conventions of spatial order. If the left or
145
Topography of Movement
Ruth Schnell
und or screen, but as an element of a new image formation.”
B
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
146
C
right hand point to an absent body, a subjective viewer perspective is made impossible by the high magnification of the image content. The scaling changes the spatial reference system. The viewer hears the original sounds that arose during the recording simultaneously to the motion of the image: wiping, knocking, patting sounds. The sound thus contributes to the spatialization of the image content. With this environment, Ruth Schnell is continuing her artistic research in the field of dynamic projection. Hands and their movements were already addressed in two previous works: Gegen die Zeit, originally conceived in 2001 for the secularized Johanneskirche in Feldkirch and B Territorism (2002), facade projection commissioned by Kunsthaus Bregenz, Bregenz Photo: Ruth Schnell C Gegen die Zeit (2001), installation, Johanniterkirche, Feldkirch Photo: Ruth Schnell
shown again at the 2008 Tiefenrausch exhibition in Linz, projected a woman’s arm with a scrubbing brush and presented a textual ambiguity between “covering one’s tracks” and “critique of reproductive work.” The multi-piece video projection Territorism, developed for the Kunsthaus Bregenz (KUB), displayed projections on the glass entrance and side facade of the KUB and the facade of the neighboring Landestheater each night in the summer of 2002. Two hands were projected; one held a toy tank, the second seemed to rest on an illusory floor. This gave rise to a simulated spacial structure that intertwined the solitary building of the KUB with its architectural environment. (Patricia Köstring / Ruth Schnell)
147
IN_SIDE VIEW Jeffrey Shaw / Sarah Kenderdine
“ In_side a pataphysical allegory for the multi-sensory
Jeffrey Shaw / Sarah Kenderdine
Artistic media: Immersive installation Primary sensory modalities: Orality (Taste), Touch, Vision, Kinesthetics 2015
IN_SIDE VIEW
A
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
148
encounter between the body and the perceptual imaginary, between nature, manufacture and ataxia.”
A Panoramic photos of Angkor Wat: Jeffrey Shaw and Sarah Kenderdine Panoramic 3D photos of a fire-damaged hardware store in Sydney: Jeffrey Shaw and Volker Kuchelmeister
“The huge snake roots, the vaster / serpent arms fallen / octopus over the roof / in a square courtyard – curved / roofcombs looked Dragon-back- / stone-scaled / As frail as stone is, this harder wooden / life crushing them // with the cricket-glare and parrot / squads walking across the roof / – last nite full moon in misted heaven / and slow girl dance bent elbow and insp[i]ring / fingers snaking it thru the middle –“ (Allen Ginsberg: “Angkor Wat”, 1963) 1 https://en.wikipedia.org/wiki/ Circle_of_confusion
“In forging the narrative of things, senses are separated to make sense of the myriad impressions” (Virginia Woolf). In IN_SIDE VIEW senses are senselessly conjoined, the world put back together, “an incessant shower of innumerable atoms” (ibid), seeing oneself as another, in side out, in the circle of confusion1, amongst trees / stone-scaled / snake roots // eyed / tongue in cheek / switchback // past / future! / present // cable / tie // wrap / around // heady / giddy / techy / tacky / vroom // memory / aromatic / terror // ruination / fervid bedlam. The where-with-all in_side a
Application software: Leith Chan Hardware: Samsung Gear VR, Conceptus tongue switch
pataphysical allegory for the multi-sensory encounter between the body and the perceptual imaginary, between nature, manufacture and ataxia.
149
Transmission+ Interference David Strang / Vincent Van Uffelen
Transmission + Interference
David Strang / Vincent Van Uffelen
Artistic media: Sound-light installation Primary sensory modalities: Vision, Audio, Kinesthetics 2016
A
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
150
“What if one could utilize someone’s unique sensing
This installation was created by David Strang and Vincent Van Uffelen and is part of their larger collaborative project Transmission+Interference. The artists explore the creative potential within light as both the creator and transmitter of sound and actively move away from creating visualizations of sound towards an audiovisual experience based on resonance and feedback loops between light, sound and object. Using performances, workshops, and installations they attempt to learn about information, disturbance, noise, glitch, communication, as well as the ephemeral relationships that emerge from both their and the audience’s engagement. The artists aim to reveal the complexity of these fleeting experiences that occur in the moment, those glitches that happen somewhere between physical sensors and mental representation that can’t be recorded, repeated, or simulated. Transmitter > Multi-Modal Crossover > Receiver Consisting of three parts, the interactive installation investigates the potential for using digital devices to create a synesthetic communication system. The installation draws on the different experiences of synesthetes and uses this to explore a messy and playful method of communication. Using digital protocols of ASCII and frequency conversion, letters are A LED and solar cell sound transmitter and receiver, Workshop in Plymouth 2013
encoded into a combination of a light color and a sound frequency (e.g., “A” is transmitted as a yellow light and a 50 Hz sine wave) allowing a simulated synesthete to decode the text that has been transmitted through space. However, to get closer to a synesthetic communication process, the classic Transmitter –> Receiver pattern must be interrupted by a multi-modal crossover that mimics a synesthete’s unique experience of light and sound. The visitor is able to adjust the multi-modal crossover to gain further understanding of a synesthete’s unique experience whilst interfering with a potentially flawless communication flow. By expanding the communication system across three distinct sections, the installation develops defined territories or zones through and
151
g ability? Would this bE AN EfPectiVe d00l T0I sTckr3ly c$m_uEicR7s!”
Light amplitude modulator
Arduino
Potentiometers
Mini Computer
Amplifier & Speaker
Micro Computer
Microphone
LEDs Colour Sensor Screen
TEXT TO SEND
Screen
Solar Cell
RECEIVER
Transmission + Interference
B
David Strang / Vincent Van Uffelen
Micro Computer
CROSS-MODAL MODULE
TEXT RECEIVED
TRANSMITTER
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
152
C
over which information (noise and signal) flows. Although these sections of the system appear as physical material in the space, the artists are actively exploring the spaces between the physical parts. It is here that we encounter the resonance within the system; where neighboring sections meet through the interference of external noise (light and sound) and through which new meaning potentially begins to emerge. The system encourages a nomadic movement of information across a technological plateau that is initially understood as linear, allowing for the spreading, leaking and disruption of information across undefined routes. What if one could utilize someone’s unique sensing ability? Would this be an effective tool to securely communicate? After all, one could covertly send hidden messages in plain sight that only an intended receiver could decode (steganography). Such a system of transmission could, in the odd event, be learnt and understood by someone able to perceive the blurring of these senses and, with practice, the user could claim our proposal as their own efficient, albeit abstract, working communication tool. B Schematic of the Transmission + Interference installation
Furthermore, the installation is a cybernetic experiment tapping into the unknowable and becoming. The physical installation is also an enquiry into practice in its own right and aims to raise questions about our relationship with technology, our co-evolution, and the blurry lines that
C Laser light sound transmission, Performance Athens 2013
our minds try to define as borders between the self, the body and other technological systems.
153
I am Sound Tamiko Thiel / Christoph Reiserer
Tamiko Thiel / Christoph Reiserer
Artistic media: Interactive installation Primary sensory modalities: Vision, Audio, Kinesthetics 2016
I am Sound
A
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
154
“I see. I hear. I am. I am
On “I am Sound” The installation is an embodied sensory experience, explain Tamiko Thiel and Christoph Reiserer, co-authors of I am Sound. The collaborators first worked together in 2003, creating a real-time sound/video improvisation, secret movements, which involved several musicians, surveillance and infrared cameras, and a live projection behind the musicians. Both artists have continued to develop their respective practices and are recognized for their innovative approach to experimental media. Thiel has produced internationally recognized works in the virtual realm, especially interactive 3D and AR (Augmented Reality) environments. Reiserer is an accomplished composer, performer and creator of sound installations. He has continuously stretched the boundaries of sound and image by using computer software and electronics. Recognizing the relationship between sound and image, and the influence this element has had on the moving image, these artists have gone beyond the typical strategy of creating a soundscape to accompany and enhance an image (or vice versa), in order to join sound and image in a unified statement. They use the image itself as the source of sound, and that image is obtained by data sourced from the viewer’s face. This A I am Sound, test of installation lighting. Tamiko Thiel and Christoph Reiserer, 2016 Mixed media installation (custom metallophone, video, audio, electronics). Dimensions: 4 m wide x 3 m high x 7 m deep (approx.).
informational interaction, the bringing in of the viewer into the work as an essential element, produces a new set of opportunities. The viewer, who is the source of what is seen and heard, is the main component of a finely tuned set of factors. To understand the dynamic of I am Sound, it’s important to appreciate that the installation is in fact a musical instrument system. Presented as a darkened room with a pool of light indicating where the visitor should stand, it is composed of three vital parts: hardware, software and the viewer. The hardware consists of a “curtain” of 12 aluminum plates which are tuned to vibrate at frequencies which produce specific tones and their harmonics. The plates function as both loudspeakers and a projection
155
Tamiko Thiel / Christoph Reiserer
am Image. I am Sound.”
I am Sound
B
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
156
C
screen. A small camera and microphone captures information from the viewer; a willing participant who, facing the camera and microphone, looks forward to where their face is projected in front of them onto the plates. The visible image is how the machine interprets the visual information received: grey, pixelated, in jittery motion, fragmented and framed by green lines as the algorithms select and transform this mirror of the self B I am Sound, shadow test of prototype #1. Tamiko Thiel and Christoph Reiserer, 2016 Mixed media installation (custom metallophone, video, audio, electronics). Dimensions: 4 m wide x 3 m high x 7 m deep (approx.). C I am Sound, metallophone prototype #2 back view: transducers and brackets. Tamiko Thiel and Christoph Reiserer, 2016 Mixed media installation (custom metallophone, video, audio, electronics). Dimensions: 4 m wide x 3 m high x 7 m deep (approx.).
into data. Converted into data, the face, captured as a lowresolution surveillance image, is digital information which vibrates, emitting unique, harmonious audio created specifically from the data obtained from each individual facing the camera. What is revealed, an augmented, amplified selfie portrait, becomes a personal musical “voice” of the subject with interconnected harmonic tones. Underpinning this work is a sense of reflection. Even though it is presented as a challenge, it is not a game but an opportunity for information to be experienced differently. The face, one’s individual identity, is typically small and exposed to the world. But this selfie is large, and seen directly by the participant who can alter and explore new and varied compositions. I am Sound invokes the multiple modes of visual, aural, kinesthetic and spatial senses and generates a contemplative customized musical and visual encounter with one’s self. (Kathy Rae Huffman)
157
Data Music Peter Weibel
Data Music
Peter Weibel
Artistic media: Interactive audio-visual installation Primary sensory modalities: Touch, Audio, Vision 2016
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
158
“Digital music turns every object sources. Each object can simulat
A notation is a fixed instruction written on sheets of manuscript paper. It is a kind of analog programming that tells a human being what to do; e.g., touch the black and white keys of a piano in a prescribed way. Whereas a program tells a machine what to do, a notation tells a human being what to do. These notational commands are written in arbitrary signs for interpretation. In contemporary music the interpreter is given more freedom, more degrees of freedom of interpretation. Today, in the age of digital music, notation has changed its character tremendously. The composer tells the computer in a prescribed way what to do, he writes a script for the computer which becomes an interpreter with a lesser degree of freedom. Digital music deals with data instead of tones and notes. The axiom of digital music is: any word, any sign, any image, any number, any tone can be transformed into data and any data can be transformed into any numbers, sounds, images and texts. Under these circumstances notation no longer serves as only an instruction but becomes an instrument. In a traditional musical set-up, a music stand – which is an empty frame made of iron or wood, a device to hold sheets of manuscript paper on which notes are written or printed – is placed in front of the musician. The musician, or alternatively the interpreter, reads these notes as commands telling him how to perform his instrument. The classical performance therefore consists of four parts: the instrument, the music stand, the performer and the invisible composer present by the notation. In the digital age, however, the notation becomes a physical process, a kind of operative ontology. Based on
159
Data Music
Peter Weibel
ect into a musical instrument. Digital music turns any object into sound late each instrument. The computer becomes a universal composer.”
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
160
digital logic, I constructed an interactive installation. The set-up comprised a music stand shaped like an empty frame with five staves. These five staves consist of five horizontal strings building a kind of abstract geometrical guitar (in a cubist way). The strings are attached to a computer in which several musical compositions are stored, e.g., famous rock riffs. Loudspeakers are furthermore connected to the computer. This setup enables the visitor to pull these five strings, which are the “notation” and the “instrument” at the same time, whereby he becomes the “composer”, “interpreter”, and “performer” of the music. As the music is stored physically as data in the computer, i.e., pre-programmed, we could say that the visitor performs music through an interface (music stand), which is data-programmed by an algorithm. In addition, the performance of the player and of the visitor respectively is recorded by a KINECT camera and projected onto a screen, or alternatively displayed by a monitor. As the images of the performance are purely data, the configuration of the visual data can easily be influenced by the configuration of the acoustic data. Sound and image are synchronized with each other. Thus, whenever the visitor is in front of the music stand, he sees himself projected or displayed in a normal, mirror-like way. The very moment the visitor starts performing with the music stand, his image becomes distorted and appears as a flow of visualized acoustic data. Classical synesthesia is a phenomenon organized by the human brain and is a subjective experience. Digital synesthesia is an objective phenomenon, organized by machines and media.
161
Data Music
Peter Weibel
This book chapter introduces 15 DIGIT been produced by 17 renowned media arts-based research process. The digit mental set-ups for the artistic research Texts, sketches and pictures provide in of the works in the research context o
Software development: Nikolaus Völzow Sound design and interface design: Derek Hauffen
ARTISTIC RESEARCH ON DIGITAL SYNESTHESIA
162
TAL SYNESTHESIA artworks which have a artists in the course of a three-year tal artworks have to be seen as experih of cross- and multi-modal perception. nformation about the artistic concepts of DIGITAL SYNESTHESIA.
163
DIGITAL SYNESTHESIA EXHIBITION Part 3
This book chapter contains the photog SYNESTHESIA exhibition at the Angew which took place from March 11 to Ap which were commissioned by the Digi the international DIGITAL SYNESTHES
164
graphic documentation of the DIGITAL wandte Innovation Laboratory in Vienna, pril 8, 2016. On display were 14 artworks ital Synesthesia Group in the course of SIA Call in 2014.
165
DIGITAL SYNESTHESIA EXHIBITION
166
A Anke Eckardt VERTICAL 2
167
Alan Kwan The Flying Umbrella Project
DIGITAL SYNESTHESIA EXHIBITION
168
Peter Weibel Data Music
169
Ruth Schnell Topography of Movement
DIGITAL SYNESTHESIA EXHIBITION
170
Marcello Mercado Bestiary for the Minds of the 21 st Century: Genomic Opera
171
R uth Schnell MotU #4–#6
DIGITAL SYNESTHESIA EXHIBITION
172
Karl Heinz Jeron Space Time
173
Jeffrey Shaw/Sarah Kenderdine IN_SIDE VIEW
DIGITAL SYNESTHESIA EXHIBITION
174
Karen Lancel / Hermen Maat E.E.G. KISS
175
Anke Eckardt VERTICAL 2
DIGITAL SYNESTHESIA EXHIBITION
176
Ulla Rauter Sound Calligraphy
177
Tamiko Thiel / Christoph Reiserer I am Sound
DIGITAL SYNESTHESIA EXHIBITION
178
kondition pluriel At Play
179
David Strang / Vincent Van Uffelen Transmission+Interference
DIGITAL SYNESTHESIA EXHIBITION
180
181
This book chapter contains the photog SYNESTHESIA exhibition at the Angew which took place from March 11 to Ap which were commissioned by the Digi the international DIGITAL SYNESTHES
Ruth Schnell facades
DIGITAL SYNESTHESIA EXHIBITION
182
graphic documentation of the DIGITAL wandte Innovation Laboratory in Vienna, pril 8, 2016. On display were 14 artworks ital Synesthesia Group in the course of SIA Call in 2014.
183
184
186 – 197 Authors’ Biographies 198 – 199 Imprint
185
Authors’ Biographies
Gerald Bast Following graduation in Law and Economics at the University of Linz, completed a doctorate in Law (1974). Head of the Federal Ministry of Science and Research Department for Organizational Law and Principle Issues regarding the reform of universities and art colleges (1991– 1999) and consultant for the Ludwig Boltzmann Society for Scientific Research (1992– 1999). Rector of the University of Applied Arts Vienna (since 2000); deputy chairman of the Federation of Austrian Universities; spokesman for the rectors of the Austrian Universities of Art, and vice-president and member of the presidium of the Conference of Austrian Universities. Member of the scientific advisory board of the “Zeitschrift für Hochschulrecht, Hochschulmanagement und Hochschulpolitik” and member of the board of the European League of Institutes of the Arts (ELIA).
Anke Eckardt Anke Eckardt’s work encompasses sculptures/installations on an architectural scale, academic research and teaching. She has developed a series of artworks on the theme of SONIC SPACES focusing on architectural elements. She mainly works with sound (including ultrasound and infrasound) and light in combination with other materials such as water and concrete, pneumatics and motors. In 2014 she published a monograph on SONIC SPACES at Revolver Publishing, Berlin. She has received grants and prizes; her artistic work is presented internationally (Ars Electronica, NEMO, Steirischer Herbst, Club Transmediale, EMAF). Besides her solo work, Eckardt is extending collaborative projects e.g., with choreographer Shannon Cooney and artist and researcher Toby Heys, light artist Ulf Pedersen, and sculptor Sandra Peters. Since 2014 Eckardt has been Artistic Associate at the Department of Visual Arts & Architecture within the School of Architecture, Urban Planning and Landscape Design at the University of Kassel, Germany. There she also undertakes her research on sound and architecture as part of the transdisciplinary platform Building Art Invention. www.ankeeckardt.org
Katharina Gsöllpointner Katharina Gsöllpointner is a media art theoretician with a special focus on media aesthetics and the cybernetics of art. She has taught media and art studies at international universities and currently works as a researcher at the Dept. of Digital Art at the University of Applied Arts Vienna. Since the 1990s Gsöllpointner has conceived, led and carried out a series of inter- and transdisciplinary research projects on issues of media aesthetics and media arts, e.g., AESTHETIC KNOW-HOW. Language – Technology – Media (2007– 2009) or Digital Synesthesia (2013– 2016). From 1991 to 1995 she was manager of the Ars Electronica Festival in Linz (w / Peter
186
Weibel). Since the 1980s she has published extensively about media and digital art history, media aesthetics and on the transdisciplinarity of art and sciences. In her habilitation treatise Intermedia Production and Multimodal Perception. On the Aesthetics of Digital Art (2015), she formulated a media theory of digital art as an aesthetic model for the multimodality of perception. www.katharinagsoellpointner.at
Kathy Rae Huffman Kathy Rae Huffman is a freelance curator, networker, writer and media art collector. She has held significant curatorial posts at the Long Beach Museum of Art, The ICA Boston, and the former Cornerhouse, Manchester (UK). She was Professor of Electronic Arts at Rensselear Polytechnic Institute, Troy, NY. For three decades, she has consulted, presented special programs, curated, juried, administered and coordinated events for numerous international media art festivals and arts organizations. Her research topics have focused on artists’ television, video art, and feminist strategies in online environments. Huffman co-founded FACES: Gender / Technology /Art, an online community for women in 1997. Huffman was awarded an MFA (summa cum laude) in Exhibition Design and Museum Studies from the School of Fine Arts, California State University, Long Beach. From 1990 – 2013, Huffman lived and worked in Vienna, Manchester and Berlin, and traveled extensively in Central and Eastern Europe. She currently resides in Southern California.
Karl Heinz Jeron Karl Heinz Jeron is a German artist living in Berlin whose practice is idea-based and focuses on the treatment of social issues. He uses customized small robots and references from his personal experience to create situations that can disrupt preconceived notions and challenge the position of the viewer. Recent exhibitions, events and collaborations include the ZKM Karlsruhe, Ars Electronica Linz, Documenta X, ICA London, Walker Art Museum Minneapolis, Berlinischen Galerie Berlin and the Museum of Modern Art San Francisco. www.jeron.org
187
Sarah Kenderdine A former maritime archaeologist, Professor Sarah Kenderdine researches at the forefront of interactive and immersive experiences for museums and galleries. In widely exhibited installation works, she has amalgamated cultural heritage with new media art practices, especially in the realms of interactive cinema, virtual and augmented reality and embodied narrative. At the University of New South Wales she is director of Expanded Perception and Interaction Centre (EPIC) and the Laboratory for Innovation in Galleries, Libraries, Archives and Museums (iGLAM). In the last 10 years she has created over 80 exhibitions including a multi-award-winning new museum in South India. She’s authored six books and later this year will complete a co-authored monograph Theorizing Digital Cultural Heritage for a Complex, Turbulent and Entangled World with MIT Press. www.niea.unsw.edu.au
kondition pluriel (Martin Kusch / Marie-Claude Poulin) Martin Kusch studied art history, philosophy and painting in Berlin, and media arts with Peter Weibel in Vienna. He is the founder and director of the Dome-lab and the project leader of the European Mobile Dome Lab for Artistic Research (E / M / D /L), at the Dept. for Digital Art, University for Applied Arts Vienna. Founder and artistic co-director of the media performance group kondition pluriel, he is particularly interested in the transformation processes of the electronic media from within performative contexts, and on how digital technologies influence our perception of the body and space. His works have been presented at numerous festivals and institutions, such as: Ars Electronica (Linz), CYNETart (Dresden), CDA (Enghien les Bains), EMPAC (Troy), FIND (Montreal), ICA (London), ISEA (Nagoya, Helsinki, Essen), SAT (Montreal), Transmediale (Berlin), Mois-Multi (Quebec City), ZKM (Karlsruhe) and Museumsquartier (Vienna). Born in Québec, Canada. Marie-Claude Poulin is a performance researcher, choreographerperformer and interdisciplinary artist who specializes in the combination of dance and new technologies. Between 1985 and 2000, she worked as a performer, notably with choreographers Benoît Lachambre and Meg Stuart / Damaged Goods. Co-founder of the media-performance group kondition pluriel (2000), she has created ground-breaking artistic experiences and performances using a wide range of cutting-edge tech tools at the intersection of dance, somatic practices and interactive installations. Marie-Claude Poulin has presented performances and installations, participated in symposia and conferences and conducted creative workshops around the world. Since 2013 she has been a lecturer at the University of Applied Arts Vienna, where she collaborates on international research projects such as E / M / D/ L and Digital Synesthesia. www.konditionpluriel.org / www.emdl.eu
AUTHORS’ BIOGRAPHIES
188
Patricia Köstring Patricia Köstring is a Vienna based publicist and cultural manager. In the 1990s she ran the Köstring/Maier gallery in Munich, an artspace dedicated to contemporary minimalism and site-specific artistic practice (together with Stephan Maier). Since she moved to Austria in 1998 she has been working in the fields of contemporary art as well as cultural politics. Currently she is a member of the editorial board of the cultural politics reader kamion and Senior Artist at the Dept. of Digital Art at the University of Applied Arts Vienna.
Alan Kwan Alan Kwan is a digital media artist and researcher. His overarching interest is in cross-disciplinary understandings of cinema and performance as well as new forms of artistic language and storytelling enabled by virtual reality and flying machines. He has presented widely as a speaker and artist-in-residence. His projects have been shown at exhibitions around the world, including the Ars Electronica Festival in Austria, ZKM Centre for Art and Media in Germany, and Museum of Contemporary Art (MOCA) Shanghai, and were also featured in major media including Discovery Channel, WIRED, Popular Science and Boston Globe. In 2014, he received the Hong Kong Arts Development Council Award for Young Artist (Media Art), and the Asian Cultural Council Fellowship to pursue his master’s degree at the Massachusetts Institute of Technology (MIT), where in 2015 he was awarded first place in the MIT Harold and Arlene Schnitzer Prize for his body of work combining art and technology. wwww.kwanalan.com
Karen Lancel / Hermen Maat Artists Lancel and Maat create meeting places in city public spaces and the digital domain. Their seductive visual performances and installations include objects, video, prints, drawings, networked databases, projections and digital networks. Audience members are invited to “artistic social labs” as “co-researchers”. In carefully designed and hosted meeting rituals, they reflect on their perception of the smart city and their experience of body, presence, identity, privacy, reciprocity and trust. In their works the artists research contemporary social systems and “trust” systems. They deconstruct and alter control and self-quantifying technologies in order to facilitate new intimate meeting experiences; allowing experimental, sensory syntheses. Audience participation is digitized, translated and printed into “social portraits of living in a networked society”. Shows: Ars Electronica, Linz; ZKM, Karlsruhe; Rijksmuseum Amsterdam; Shanghai World Expo 2010, DDC / Mobile City; Venice Bienniale 2015 / Chinese Pavillion; Public Art Lab Berlin –
189
Connecting Cities; Stedelijk Museum Amsterdam; Banff Center Canada; Art Center Nabi Seoul South-Korea; Eyebeam New York; ISEA Hong Kong 2016; ISEA2011 Istanbul; ISEA04 / Kiasma Museum Helsinki; Urban Screens08 Melbourne; NIMK Amsterdam; V2_Institute Rotterdam; Transmediale Berlin; Utrecht University Play, Perform, Participate; Beijing Contemporary Art Center (BCAC); Holland Festival Amsterdam / De Balie; Frascati Theatre Amsterdam; Eye Film Institute Amsterdam; De Appel Amsterdam; TASIE-2012 TsingHua University, Science & Technology Museum Beijing; TASIE-2006 Millennium Museum Beijing. Karen Lancel is an artistic PhD candidate at Technical University of Delft: “Participatory Systems Initiative”; she was a member of the AHK Amsterdam research group “ARTI” (Artistic research, Theory & Interpretation) and headed the interactive media art department, MFA Frank Mohr Institute Groningen 2005-2008. Hermen Maat teaches media art at the Minerva Art Academy Groningen, and is a member of the research group on Image in Context. www.lancelmaat.nl
Geert Lovink Geert Lovink is a Dutch /Australian media theorist, internet critic and author of Zero Comments (2007), Networks Without a Cause (2012) and Social Media Abyss (2016). Since 2004 he has been a researcher at the School for Communication and Media Design at the Amsterdam University of Applied Sciences (HvA), where he is the coordinator of the Institute of Network Cultures (INC). His center recently organized conferences, publications and research networks such as Video Vortex (the politics and aesthetics of online video), Unlike Us (alternatives in social media), Critical Point of View (Wikipedia), Society of the Query (the culture of search), MoneyLab (bitcoins, crowdfunding & internet revenue models) and the future of art criticism project De Kunst van de Kritiek. In 2015 the INC digital publishing research was split off into the Publishing Lab. Since 2009 he has been a professor at the European Graduate School (Saas-Fee) where he supervises PhD students. www.networkcultures.org
Lawrence E. Marks After receiving a PhD in experimental psychology from Harvard University in 1965, Lawrence Marks moved to Yale University, where he is currently professor of environmental health sciences and psychology, and to the affiliated John B. Pierce Laboratory, where he is Fellow and Emeritus Director. Marks’s research deals broadly with the processes and phenomenology of human perception: In particular, Marks’s work emphasizes the study of multisensory perception – the mechanisms by which signals from multiple sensory modalities combine and interact
AUTHORS’ BIOGRAPHIES
190
to produce perceptual experiences – and in the ways that multisensory perceptual information is coded in language. For more than four decades, Marks has studied synesthesia in perception and language, including the role of synesthesia in metaphor. Marks is the author or co-author of more than 200 journal articles, book chapters, and conference proceedings, as well as two books, including The Unity of the Senses: Interrelations among the Modalities. www.jbpierce.org
Marcello Mercado Biological theories, mathematical principles, and technology are founding elements in Mercado’s work. His interdisciplinary projects create a dialogue between visual art, robotics, bio-art and music. In 1996 he was Fellow at The John D. and Catherine T. MacArthur Foundation, The Rockefeller Foundation and The Lampadia Foundation and in 1999 Fellow at The Academy of Media Arts Köln; in 2000 Post-Production Residence at the Centre International du Creation Vidéo Pierre Schaeffer, Montbéliard, France. He received the Grant for Media Art 2013 from the Foundation of Lower Saxony at the Edith-Russ-Haus for Media Art. Mercado was awarded with the Honorary Mention, Prix Ars Electronica 2012 in Interactive Art; He has exhibited in many museums, galleries and festivals, amongst others at the ZKM Karlsruhe; Biennale di Venezia; Centre Pompidou; The Taipei Fine Arts Museum; Museo Nacional Centro de Arte Reina Sofía; 11th Biennale de l’Image en Mouvement, Genéve; Kunsthaus Dresden; Museo de Arte Moderno de Buenos Aires. www.marcellomercado.com
Aleksandra Mroczko-Wąsowicz Aleksandra Mroczko-Wąsowicz is currently Associate Professor of Philosophy and Cognitive Science at the National Yang-Ming University in Taipei, Taiwan. Her research centers on philosophy of mind, philosophy of perception, philosophy of psychology and cognitive science, neurophilosophy, consciousness, and synesthesia. She received her Ph.D. in Philosophy from the University of Mainz in 2011 with her dissertation on The Unity of Consciousness and the Phenomenon of Synesthesia. Aleksandra Mroczko-Wąsowicz has published articles on synesthesia and other topics in cognitive science and philosophy of perception in the Oxford Handbook of Synesthesia, Open MIND, Consciousness and Cognition, Cortex, Frontiers in Human Neuroscience, Journal of Vision, and Frontiers in Psychology. She is an associate editor of Theoretical and Philosophical Psychology and handling editor of the Research Topic Perception-Cognition Interface & Cross-Modal Experiences: Insights into Unified Consciousness at Frontiers in Consciousness Research. www.mroczko-wasowicz.com
191
Catherine M. Mulvenna Catherine Mulvenna received an MA in experimental psychology in 2003 and an MSc in psychological research in 2004, both from the University of Glasgow; followed by an MPhil in 2006 and a PhD in 2012, both in cognitive neuroscience and psychology, from University College London. Subsequently, Mulvenna held postdoctoral positions at Yale University before returning, in 2013, to the University of Glasgow where she is now training at the Institute of Health and Wellbeing. The author or co-author of ten articles and book chapters, Mulvenna’s research has focused on the neural and psychological mechanisms that underlie synesthesia, with special interest in the connections between synesthesia and creativity. www.scn.ucla.edu/people/mulvenna.html
Danko Nikolić The main motive for my studies is the explanatory gap between the brain and the mind. My interest is in how the physical world of neuronal activity produces the mental world of perception and cognition. I am associated with the Max-Planck Institute for Brain Research, the Ernst Strüngmann Institute, the Frankfurt Institute for Advanced Studies, and the University of Zagreb. I approach the problem of explanatory gap from both sides, bottom-up and top-down. The bottom-up approach investigates brain physiology. The top-down approach investigates behavior and experiences. Each of the two approaches has led me to develop a theory: The work on behavior and experiences led me to discover the phenomenon of ideasthesia (meaning “sensing concepts”). The work on physiology resulted in the theory of practopoiesis (meaning “creation of actions”). www.danko-nikolic.com
Regine Rapp Regine Rapp is an art historian and curator. Her specific fields of research for 20th and 21st century art are installation art, image text theory, artist books, and art & science collaborations. She worked as Assistant Professor for Art History at the Burg Giebichenstein Art Academy Halle until autumn 2013. She is co-director of Art Laboratory Berlin, which she co-founded in 2006. She curated over 30 exhibitions (e.g., the series Time and Technology, and Synaesthesia) and has published several books. She conceived and realized the international Sol LeWitt_ Symposium at Art Laboratory Berlin. She developed the international transdisciplinary conference Synaesthesia. Discussing a phenomenon in the arts, humanities and (neuro-)science, 2013 at Art Laboratory Berlin. Her new publication [macro]biologies & [micro]biologies. Art and the Biological Sublime in the 21st Century theoretically reflects Art Laboratory Berlin’s last series from 2014 –15. www.artlaboratory-berlin.org
AUTHORS’ BIOGRAPHIES
192
Ulla Rauter Ulla Rauter studied Transmedia Art at the University of Applied Arts Vienna. As a media artist and musician, she works at the interface between sound and fine art – her works include performative sculptures, music performances and self-built instruments. In 2009 she received a working scholarship from the Austrian Federal Ministry for Science and Research; in 2011 a start-up scholarship from the Austrian Federal Ministry for Education, the Arts and Culture. Exhibitions (selection): Silent roomnumberOne, Mario Mauroner Contemporary Art Vienna 2015 | Erwartung, Projekt Fabrika, Moscow 2014 | Out of the Box, MAK Vienna, 2013 | MEX21, Künstlerhaus Dortmund, 2013 | EAR ZOOM Sonic Arts Festival, Ljubljana 2013 | ART, SCIENCE AND SOCIETY / 12th ELIA Biennial Conference, Vienna 2012 | on to red, Galerie der Künstler, Munich 2012 | SIGNS & SIGNALS, Ars Electronica, Linz, 2009 | Austria for Beginners, Museum of New Art, Detroit 2009 www.ullarauter.com
Christoph Reiserer Christoph Reiserer studied musicology, music education and philosophy in Munich and Berlin and took private lessons in composition with Stefan Zorzor. He has been producing his own projects since 1998; for example up & down (2001, Deutsches Museum Munich), so far (2003 as artist-in-residence at the CCA Glasgow, UK) and fluX (2008 for the 850th anniversary of the City of Munich). His recent work increasingly incorporates more video and electronics, including the sound installation tisch (2005), U-Musik (2006, Munich subway stations) and tunnel ende (2010, Klangspuren Schwaz, Austria). He composed three chamber operas: Und wenn wir dann soweit sind, können wir anfangen (2007), Die Nacht des Brokers (2010) and President Jekyll (2012). For the Münchener Biennale he realized the concert installation some work (2010, Museum Villa Stuck). Awards for his work include the Leipzig Improvisation Competition in 1997 and the Music Prize from the City of Munich in 2009. www.christophreiserer.net
193
Chris Salter Chris Salter is an artist, University Research Chair in New Media, Technology and the Senses at Concordia University, Co-Director of the Hexagram network for Research-Creation in Media Arts, Design, Technology and Digital Culture in Montreal and Interim Co-Director of the newly-founded Concordia Institute for Digital Art, Culture and Technology. He studied philosophy and economics at Emory University and completed a PhD in directing and dramatic criticism at Stanford University where he also researched and studied at CCMRA. His work has been seen all over the world at such venues as the Venice Architecture Biennale, Vitra Design Museum, BIAN 2014 (Montreal), LABoral, Lille 3000, CTM Berlin, National Art Museum of China, Villette Numerique, Todays Art, Mois Multi, Transmediale, EXIT Festival (among many others. He is the author of Entangled: Technology and the Transformation of Performance (MIT Press, 2010) and Alien Agency: Experimental Encounters with Art in the Making (MIT Press, 2015). He is currently working on a new collaborative book project for the MIT Press focused on the contemporary interconnections between art, theory and research. www.chrissalter.com
Ruth Schnell Ruth Schnell is a media artist based in Vienna. Since 2010 she has been Head of the Digital Art Department at the University of Applied Arts Vienna. She has been working with computer-aided tools since the mid-1980s. Her corpus of work, which includes video installations, interactive video environments and light installations, explores the nature of human perception and the relationship between human perception and the human body. She is considered an expert in dynamic projection in particular. Moreover, she has done pioneering work in programming electronic LED light sticks visualizing the after-image phenomenon in different series of works. Solo and group exhibitions include: Satosphère, Society for Arts and Technology [SAT], Montreal /CA, 2015; Moscow Biennale of Contemporary Art / RU, 2011 and 2013; ZKM / Center for Art and Media Karlsruhe / DE, 2010 and 2008; Bienal Internacional de Arte Contemporáneo de Sevilla / BIACS 3 / ES, 2009; Akademie der Künste Berlin / DE, 2004/05; California Science Center, Los Angeles / USA, 2004; Kunsthaus Bregenz / AT, 2002; mumok – Museum moderner Kunst Stiftung Ludwig Wien / AT, 2000; 46. Biennale di Venezia / Austrian Pavilion / IT, 1995. www.ruthschnell.org
Romana K. Schuler Romana K. Schuler, art historian, wrote her dissertation on experimental perception in the arts and sciences (“Seeing Motion“ 2015). The emphasis was on oscillopsia in virtual reality. She has researched, written, taught
AUTHORS’ BIOGRAPHIES
194
and curated on new media in contemporary art since 1987. She conducted research on Austrian media art from 1992 to 1994 with an emphasis on the concept of sculpture in digital media and from 2013 to 2016 within the arts-based research project “Digital Synesthesia“. From 1995 to 2005 she was involved in the development of the Leopold Museum, Vienna. Since 2010 she has lectured on the history of experimental perception in the arts and sciences at the University of Applied Arts Vienna. In 2015 she co-curated the exhibition “Contemporary Code — Artistic Research“ at the City University of Hong Kong. www.romana-schuler.at
Jeffrey Shaw Jeffrey Shaw has been a leading figure in new media art since the 1960s, and in a prolific oeuvre of widely exhibited works has pioneered the use of digital media in the fields of virtual and augmented reality, immersive visualization environments, navigable cinematic systems and interactive narrative. Shaw was the founding director of the ZKM Institute for Visual Media Karlsruhe (1991– 2002), co-founded and directed the UNSW iCinema Centre for Interactive Cinema Research (2003– 2008), and since 2009 has been Chair Professor of Media Art at City University in Hong Kong where he is also Director of the Applied Laboratory for Interactive Visualisation and Embodiment. In 2014 he was appointed Visiting Professor at both the Central Academy of Fine Art, Beijing and the Institute of Global Health Innovation, Imperial College, London. In 2015 Shaw was awarded the Ars Electronica Golden Nica for Visionary Pioneer of Media Art, and in 2016 became Honorary Professor at Danube University, Krems. www.jeffreyshawcompendium.com
Miriam Spering Miriam Spering is a psychologist and neuroscientist investigating how the brain uses visual information to control movements. She is Assistant Professor in Ophthalmology & Visual Sciences at the University of British Columbia (UBC), Vancouver, Canada, and a member of the Centre for Brain Health and Institute for Computing, Information and Cognitive Systems at UBC. She obtained her graduate degree (Diplom) in Psychology from the University of Heidelberg, Germany, in 2002. She then moved on to the University of Gießen, Germany, to work with Karl Gegenfurtner for her doctoral degree in Psychology (summa cum laude, 2006). She completed a postdoctoral fellowship, funded by the German Research Foundation, with Marisa Carrasco at New York University, New York, and joined UBC in 2011. Her research has been recognized by many awards, most recently an Early Career Scholar Award by UBC’s Peter Wall Institute for Advanced Studies, an inter-disciplinary research institute. www.visualcognition.ca/spering/people.html
195
David Strang David Strang is an artist and researcher working with sound and interactive elements. His work explores the creative potential within the movement of noise in and around systems of sound and light by building / hacking bespoke devices and tools for performance, workshop, installation and intervention. Through processes of playful interaction and making, his practice investigates the links between objects, material, consciousness and the body. He currently lives and works in the UK and is a Lecturer in Music at Plymouth University. www.davidstrang.co.uk
Tamiko Thiel Tamiko Thiel is a visual artist exploring the interplay of space, the body, cultural memory and identity. She has won awards from the MacDowell Colony, MIT, WIRED Magazine, Japan Foundation and IBM Innovation Award, and exhibited internationally at venues such as the Istanbul Biennial, Art Gwangju, Corcoran Gallery of Art, Contemporary Istanbul, ICP / NY, ZKM, Tokyo Metropolitan Museum of Photography, Ars Electronica, Zero1 Biennial, SIGGRAPH and ISEA. Her work is featured in the references Digital Art by Whitney Museum curator Christiane Paul, World of Digital Art by DAM Berlin gallerist Wolf Lieser, Not Here Not There special issue of Leonardo Electronic Almanac and The Total Work of Art: From Bayreuth to Cyberspace by Stanford professor Matthew Smith. She has taught and lectured at institutions such as Carnegie Mellon University, the MIT Media Lab, the Bauhaus-University / Weimar, the USC School of Cinema-Television, the Berlin University of the Arts and Nanyang Technological University / Singapore. www.tamikothiel.com
Vincent Van Uffelen Vincent Van Uffelen works with code, hacked devices, and custom electronics to create light and sound as a medium for his artistic enquiry into process, complexity, systems, and the fleeting moments of experience. Driven by his interest in change and the entanglement of body and mind in becoming, he focuses on the performative aspects in his practice. His work also encompasses workshops, interventions, and installations. He currently lives and works in London, UK . www.vincentvanuffelen.com
Peter Weibel Chairman and CEO of ZKM | Center for Art and Media Karlsruhe and professor of media theory at the University of Applied Arts Vienna, Peter Weibel is considered a central figure in European media art on account of his various activities as artist, theoretician, and curator. He publishes
AUTHORS’ BIOGRAPHIES
196
widely in the intersecting fields of art and science. His career has taken him from studying literature, medicine, logic, philosophy, and film in Paris and Vienna and working as an artist to head of the digital arts laboratory at the Media Department of New York University in Buffalo (1984 to 1989) and founding director of the Institute of New Media at the Städelschule in Frankfurt /Main (1989– 1995). As artistic director he was in charge of Ars Electronica in Linz (1986– 1995), Seville Biennial (BIACS3, 2008), and Moscow Biennale of Contemporary Art (2011). He commissioned the Austrian pavilions at Venice Biennale (1993– 1999) and was chief curator of the Neue Galerie Graz (1993 to 2011). www.peter-weibel.at
Siegfried Zielinski Siegfried Zielinski is professor of media theory, archaeology and variantology of the media at the Berlin University of the Arts, Michel Foucault Professor for techno-aesthetics and media archaeology at the European Graduate School Saas Fee (CH), and director of the Vilém Flusser Archive in Berlin. He was Founding Rector of the Academy of Media Arts Cologne (1994–2000) and in 2015 was elected as the new Rector of the Staatliche Hochschule für Gestaltung Karlsruhe. He is the author of numerous books and articles as well as editor of the book series Variantology: On Deep Time Relations between the Arts, Sciences, and Technologies, of which six volumes have been published so far (2005–2013). Siegfried Zielinski is a member of the North-Rhine-Westphalia Academy of Sciences, Humanities, and the Arts, the Academy of the Arts Berlin, and the Magic Lantern Society of Great Britain.
197
Imprint
198
Katharina Gsöllpointner, Ruth
Concept and editing:
Schnell, Romana K. Schuler, all:
Katharina Gsöllpointner
University of Applied Arts Vienna, Photo credits:
Vienna, Austria
© artists, authors, DIGITAL Library of Congress Cataloging-
SYNESTHESIA. Exhibition photos
in-Publication data:
by Birgit and Peter Kainz, Martin
A CIP catalog record for this book
Kusch (p. 172, 179), Johnny Ranger
has been applied for at the Library
(p. 182), Nikola Tasic (p. 182)
of Congress. All images published with the Bibliographic information
kind permission of the copyright
published by the German National
holders. Although every effort was
Library:
made to obtain proper credit infor-
The German National Library lists
mation and permission to repro-
this publication in the Deutsche
duce images, the publishers would
Nationalbibliografie; detailed
be grateful to receive information
bibliographic data are available on
from any copyright holder not
the Internet at http://dnb.dnb.de.
credited herein.
This work is subject to copyright.
Texts credits: © authors
All rights are reserved, whether the whole or part of the material is
Translation, copy editing, proof
concerned, specifically the rights
reading: Andrea Lyman, Alun
of translation, reprinting, re-use
Brown, Gretchen Simms
of illustrations, recitation, broadcasting, reproduction on micro-
Graphic Design:
films or in other ways, and storage
Grafikum – Studio für Gestaltung,
in databases. For any kind of use,
www.grafikum.com
permission of the copyright owner must be obtained.
Printing:
© 2016 Walter de Gruyter GmbH,
Wolkersdorf, Austria
Berlin/Boston
Printed on acid-free paper
Holzhausen Druck GmbH,
produced from chlorine-free pulp. TCF ∞ Printed in Austria
ISSN 1866-248X ISBN 978-3-11-045934-0 e-ISBN (PDF) 978-3-11-045993-7
The artistic research project
e-ISBN (EPUB) 978-3-11-045948-7
DIGITAL SYNESTHESIA has been
www.degruyter.com
Fund FWF (2013 –2016).
www.digitalsynesthesia.net
In cooperation with School of
funded by the Austrian Science
Creative Media / City University of Hong Kong (Prof. Jeffrey Shaw)
199