Conference PaperPDF Available

Sounding Human with Data: The Role of Embodied Conceptual Metaphors and Aesthetics in Representing and Exploring Data Sets

Authors:

Abstract

Auditory display is the use of sound to present information to a listener. Sonification is a particular type of auditory display technique in which data is mapped to non-speech sound to communicate information about its source to a listener. Sonification generally aims to leverage the temporal and frequency resolution of the human ear and is a useful technique for representing data that cannot be represented by visual means alone. Taking this perspective as our point of departure, we believe that sonification may benefit from being informed by aesthetic explorations and academic developments within the wider fields of music technology, electronic music and sonic arts. In this paper, we will seek to explore areas of common ground between sonification and electronic music/sonic arts using unifying frameworks derived from musical aesthetics and embodied cognitive science (Kendall, 2014; Lakoff & Johnson, 1999). Sonification techniques have been applied across a wide range of contexts including the presentation of information to the visually impaired (Yoshida et al., 2011), process monitoring for business and industry (Vickers, 2011), medical applications (Ballora et al., 2004), human computer interfaces (Brewster, 1994), to supplement or replace visual displays (Fitch & Kramer, 1994), exploratory data analysis (Hermann & Ritter, 1999) and, most importantly for the current milieu, to reveal the invisible data flows of smart cities and the internet of things (Rimland et al., 2013; Lockton et al., 2014). The use of sonification as a broad and inclusive aesthetic practice and cultural medium for sharing, using and enjoying information is discussed by Barrass (2012). As networked smart societies grow in size and becomes increasingly complex the ubiquitous invisible data flows upon which these societies run are becoming hard to monitor and understand by visual means alone. Sonification might provide a means by which these invisible data flows can be monitored and understood. In order to achieve this type of usage, sonification solutions need to be applicable to and intelligible to an audience of general listeners. This requires a universal shared context by which sonifications can be interpreted. Embodied cognition researchers argue that the shared physical features of the human body, and the capacities and actions which our bodies afford us, define and specify mid-level structures of human cognitive processing, providing shared contexts by which people can interpret meaning in and assign meaning to their worlds (Lakoff and Johnson 1980; 1999; Varela et al., 1991). At present, embodied perspectives on cognition are infrequently explored in auditory display research, which tends to focus on either higher level processing in terms of language and semiotics (Vickers, 2012) or lower level processing in terms of psychoacoustics and Auditory Scene Analysis (Carlile, 2011).
Sounding Human with Data: The Role of Embodied Conceptual
Metaphors and Aesthetics in Representing and Exploring Data Sets
Stephen Roddy
CONNECT, Centre for Future
Networks and Communications,
Trinity College Dublin, Ireland
roddyst@tcd.ie
Brian Bridges
School of Creative Arts and Technologies,
Ulster University, Magee campus
bd.bridges@ulster.ac.uk
Introduction
Auditory display is the use of sound to present information to a listener. Sonification is a
particular type of auditory display technique in which data is mapped to non-speech sound to
communicate information about its source to a listener. Sonification generally aims to leverage
the temporal and frequency resolution of the human ear and is a useful technique for representing
data that cannot be represented by visual means alone. Taking this perspective as our point of
departure, we believe that sonification may benefit from being informed by aesthetic
explorations and academic developments within the wider fields of music technology, electronic
music and sonic arts. In this paper, we will seek to explore areas of common ground between
sonification and electronic music/sonic arts using unifying frameworks derived from musical
aesthetics and embodied cognitive science (Kendall, 2014; Lakoff & Johnson, 1999).
Sonification techniques have been applied across a wide range of contexts including the
presentation of information to the visually impaired (Yoshida et al., 2011), process monitoring
for business and industry (Vickers, 2011), medical applications (Ballora et al., 2004), human
computer interfaces (Brewster, 1994), to supplement or replace visual displays (Fitch & Kramer,
1994), exploratory data analysis (Hermann & Ritter, 1999) and, most importantly for the current
milieu, to reveal the invisible data flows of smart cities and the internet of things (Rimland et al.,
2013; Lockton et al., 2014). The use of sonification as a broad and inclusive aesthetic practice
and cultural medium for sharing, using and enjoying information is discussed by Barrass (2012).
As networked smart societies grow in size and becomes increasingly complex the ubiquitous
invisible data flows upon which these societies run are becoming hard to monitor and understand
by visual means alone. Sonification might provide a means by which these invisible data flows
can be monitored and understood.
In order to achieve this type of usage, sonification solutions need to be applicable to and
intelligible to an audience of general listeners. This requires a universal shared context by which
sonifications can be interpreted. Embodied cognition researchers argue that the shared physical
features of the human body, and the capacities and actions which our bodies afford us, define
and specify mid-level structures of human cognitive processing, providing shared contexts by
which people can interpret meaning in and assign meaning to their worlds (Lakoff and Johnson
1980; 1999; Varela et al., 1991). At present, embodied perspectives on cognition are infrequently
explored in auditory display research, which tends to focus on either higher level processing in
terms of language and semiotics (Vickers, 2012) or lower level processing in terms of
psychoacoustics and Auditory Scene Analysis (Carlile, 2011).
Roddy and Bridges, Sounding Human with Data, MusTWork 2016 64
Sonification as Data-Framing: Structural, Narrative, Aesthetic.
Broadly speaking, sonification is useful when representing/investigating data sets which embody
significant temporal variation. Sonification gives access to a temporal evolution (and can speed
up or slow down temporal processes, depending on the nature of the data sampling). This
temporal variation implies narrative and causality; the mapping of data to temporally–evolving
sound may reveal significant events through audible significant deviations within the sound. In
this way, sonification has been used within the context of data analytics using data from business
(Worrall, 2009) or socioeconomic data and may be helpful in searches for patterns within data
which may not be as apparent using visual representational strategies. The temporal and
frequency sensitivity of the auditory system may tend to make sudden (transient) differences in
the formal structure of incoming signals quite obvious, as long as basic sensitivity/just noticeable
difference (jnd) ranges are taken into account (or various temporal/frequency scale mappings are
investigated).
Fitch and Kramer (1994) provide an example of a rich mapping strategy for an auditory display
that is designed to help users monitor and respond to medical complications across eight
physiological variables of a digital patient. Two physiological variables are mapped to control
sounds to which they resemble. Heart rate was mapped to a rhythmic thudding sound and
breathing rate was mapped to a breath like sound. Atrio-ventricular dissociation and fibrillation
were mapped to modulate the heart rate sounds in the same way these factors modulate a
heartbeat in the real world. These mappings leveraged the users previous knowledge embodied
knowledge of human anatomy. Four other mappings, body temperature to a filter applied to the
heart beat sound, blood pressure to the pitch of the heart sound, brightness of the heat sound to
CO2 level and pupillary reflex to a high pitched tone, were abstract and so required learning on
the part of the listener. Empirical evaluation showed the auditory display system to be more
effective than a visual display for helping users monitor and respond to changes in the condition
of the digital patient.
Sonification practices are prevalent throughout science and the arts. This mixed appeal has led to
its use as a public outreach tool for popularising scientific research (Supper, 2014). The search
for the Higgs Boson at CERN’s Large Hadron Collider installation in Geneva is embracing
sonification as a means of public outreach 1. Researchers at NASA have are also making
extensive use of sonification as a public outreach tool2. They have also made new discoveries
using sonification, which would not have been possible through visual media alone3. Researchers
at the Climate Change Research Center used sonification techniques to create The Climate
Change Symphony in order to communicate climate change data to the public4.
Sonification is emerging as a critical technique for understanding and communicating large
complex data flows in the context of increasingly networked and data-driven societies (Rimland
et al, 2015; Hermann and Hunt, 2005). This has resulted in increased interest in sonification as a
tool for observing network activity patterns and monitoring network security, as evidenced by a
number of notable contemporary projects (see Worrall, 2015; Wolf and Fiebrink, 2013; Fairfax
et al, 2014). A number of artists have used sonification techniques to reveal the hidden data
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
"!#$$%&''$()*$+,-./$()/012'34)(1'54.$(64678$18)+$+89:128$#(85+:7!
;!#$$%.&''<<</=1>$>?(/012'<+$0#@3A-0B453C4DEF!
G!#$$%&''<<</(016124.$/012'6(<.'.04(60(8+6)8$(0#61,17=';"HIJII;8.04(6$49408)+$+8247#$8?(894,,()842%1:$+6$8
$#467.8<+4$4678?(8)4.013(:()!
J!#$$%&''<<</.04)(3/6($'7,1?+,')+$+'2>,$42()4+'0,42+$(8.=2%#16=8)+$+8.1>6)/#$2,!
Roddy and Bridges, Sounding Human with Data, MusTWork 2016 65
flows of smart cities and the IoT. Kasper Fangal Skov’s Sonic Particles 2.0 project sonifies data
provided by smart sensors placed in major cities for Data Canvas’ Sense Your City competition5.
Stanza, a UK based sound artist, makes extensive use of environmental sensor data in his artistic
sonification practices6. The Social City Detector project and the Citigram project (Park et al,
2013) used sonification to integrate the digital and physical layers of the city by making social
data visible through sound7. The Phantom Terrains project used a repurposed hearing aid to
reveal the electromagnetic signals of the wireless networks, which pervade the contemporary
built environment.8 Composers Natasha Barret and Andrea Polli make extensive use of the
sonification of environmental data in their compositional practices, often with the activist intent
of raising awareness about important environmental issues (see Barret and Mair, 2014; Polli
2012).
Beyond the arts, technology researchers are also examining the use of sonification practices to
help reveal, analyse and understand the rich data flows of the IoT. Eva Sjuve’s (2015) Metopia is
a research project that explores sonification as a medium for representing data in the context of
the IoT and Big Data. The Sound of Things project aims to add sound to the IoT. It also reveals
the invisible IoT networks of smart cities through novel applications including agogic maps9, geo
located tweet sonificaiton10. A number of hardware applications that generate live sonified sound
streams when physically attached to IoT devices have also emerged in recent years (Barrass and
Barrass, 2013; Lockton et al., 2014).
Barrass (2012) discusses how the current aesthetic turn in the field is driving the adoption of
sonification as a mass cultural medium by making sonification more appealing to the listener.
Rimland et al (Rimland et al, 2015) discusses how as societies become increasingly networked
and the IoT grows in size and complexity sonification will be needed as a means of making sense
of the complex data flows that can no longer be effectively understood by visual means alone.
Listeners will turn to sonification for enjoyment, aesthetic appreciation and to learn about the
data sources represented therein. In the coming years sonification will become a popular method
by which the invisible data flows of smart cities and smart environments are revealed and by
which the digital environment is manifest in the physical environment.
What some of these applications illustrate is that sonification may bring with it a consideration of
aesthetics; how data may be rendered in sound such that its structure is not only accessible, but
‘attractive’/’engaging’, perhaps even ‘beautiful’…certainly, sufficiently engaging to hold interest
over longer time-spans spent interrogating data sets in this way. More broadly, the sense of
narrative causality and dynamism within sonification makes it an emotive technique; data may
become more ‘impactful’, less ‘neutral’, when perceptualised. The temporal evolution and
frequency variation of sonification may be seen as corresponding to a basic model of emotion
(arousal/valence), which has previously been identified as one underpinning music’s emotional
efficacy (Huron, 2006). Technology artist Luke Dubois11 has made the point that music is an
‘emotional technology’, a data–art…the similarity relationship between sound–data–art
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
K!#$$%&''<<</-+.%(:.-138+>)41)(.476/)-'%:1L(0$.MN1640O+:$40,(.;/#$2,!
H!#$$%&''<<</.$+6P+/01/>-'(2(:7(6$04$='@$+7A.164940+$416!
Q!#$$%&''$#(0:(+$1:.%:1L(0$/340(/012'$#(82+-(:.8.(:4(.'$#(82+-(:.8.(:4(.8.104+,804$=8)($(0$1:!
R!#$$%&''%#+6$12$(::+46./012'!
I!#$$%&''<<</.1>6)19$#467./1:7'O:1L'D71740S+%./#$2,!
"T!#$$%&''<<</.1>6)19$#467./1:7'O:1L'N1649=U<(($/#$2,!
""#$$%.&''<<</$()/012'$+,-.':M,>-(M)>?14.M46.47#$9>,M#>2+6M%1:$:+4$.M2+)(M9:12M)+$+@,+67>+7(A(6!!
Roddy and Bridges, Sounding Human with Data, MusTWork 2016 66
(sonification) and music may also run in the opposite direction! As such, both (a) basic structural
framing (accessible to auditory parsing processes), and (b) narrative/affective qualities, may be
relevant considerations when exploring sonification strategies and may be unified within an
aesthetic domain.
Care must be taken when considering aesthetics in a sonification context and to this end it is
useful to draw a sharp distinction between aesthetic (for structural ends) and cosmetic
sonification practices. Cosmetic sonification practices simply aim to produce an attractive and
easily listenable sonic result while aesthetic sonification practices aims to frame and shape the
qualities of the listeners’ sonic experience as a means of communicating information about a
data source. While it is important that a sonification should sound pleasing and be easy to listen
to, especially if a listener is expected to attend to it repeatedly or for an extended period of time,
aesthetic considerations reach far beyond this concern to the framing and shaping of the listeners
very experience of the sonification, and so their understanding of the original data. It has been
argued that the aesthetic dimensions of sound are those best suited to the communication of data
in a sonification context (Roddy & Furlong 2014; Roddy, 2015).
Even if some sort of narrative of dynamic change is obvious via the audible changes within a
sonification ‘signal’, does that render the sonification inherently meaningful? How might
abstract data be converted to sound such that the structure of the data is accessible to
interpretation? There is no uniform consensus in terms of strategies; sonification designers must
currently answer questions of mapping (transposing from data structures to sound parameters)
during the design phase (Flowers, 2005). Successful sonification is not simply a straightforward
case of mapping data to isomorphic sound parameters (an approach which is basically formalist;
one which found favour in early explorations of auditory display techniques). Listeners may not
always be able to easily interpret sounds in the absence of consideration of certain perceptual–
conceptual predispositions. For example, certain dimensions within a sonification, whilst
isomorphic in terms of a dataset, may appear ‘counter–intuitive’; for example, an increase in a
dimension could be represented by a similar change in magnitude of the frequency of a tone, but
if the resulting parametric change were a decrease in frequency (albeit with similar magnitude),
the polarity of the sonification might appear to be reversed. This phenomena is explored in depth
by Walker (2000) and Roddy (2015).
The Mapping Problem and the Dimensions of Sound
The mapping problem represents a significant open problem within the field of auditory display
(Worrall 2009). It was first introduced by Flowers (2005) who criticised the central claim of
auditory display research: that submitting the contents of complex data sets to sonification will
necessarily lead to the emergence of meaningful relationships in the data. In reality, this is rarely
the case. It has been argued that the dominant conceptualisation of sound in the West is tailored
to describing the physical acoustic waveform and its perceptual correlates but that it cannot
adequately account for how sound communicates information to a listener (Truax, 1984). An
analogous argument has been made about Western art music, which reduces the rich multi-
dimensional spectra of musical and sonic possibilities to just three primary dimensions (pitch,
duration and timbre) which can be easily represented on a written score. Worrall (2010) argues
that this reductive approach to music is informed by the computationalist theory of mind and
auditory display researchers often impose these same limits upon their own work by using the
music and sound synthesis software developed for these paradigms, failing to account for the
role of the embodied performer and the perceptual and cognitive configuration of the embodied
listener. The mapping problem may be seen as the result of a tendency amongst auditory display
researchers to adopt the acoustic/psychoacoustic and Western art music paradigm when
specifying sound, thus imposing a set limits on how sound can be conceptualised, parameterised
Roddy and Bridges, Sounding Human with Data, MusTWork 2016 67
and used to communicate data to a listener. This psychoacoustic paradigm can result in auditory
display solutions that are not designed to exploit the full range of communicative dimensions
provided by sound and music and which do not account for the perceptual and cognitive faculties
of the listener.
This issue is exemplified in one of the complications resulting form the mapping problem:
dimensional entanglement. This is the intermingling of auditory dimensions traditionally
assumed to be separable within the computationalist framework. For example, in PMSon pitch,
loudness, duration and timbre are often mapped to unique data (see Grond and Berger, 2011).
However, these dimensions are not perceived independently of one another but are perceived as
individual aspects of larger sonic wholes. They are integrated and changes in one dimension can
cause changes in another making it confusing for the listener to interpret a PMSon sonification
during listening (Peres and Lane, 2005; Worrall, 2010; Peres, 2012). Ideas of discrete sonic
dimensions such as timbre, pitch and amplitude have little to do with the listeners’ everyday
experience of sound. From the perspective of embodied cognition and the ecologically–grounded
theories of perception which have influenced it, these are not meaningful dimensions along
which to sonify data. Mapping data to such parameters is not sonification but acoustification, the
straightforward mapping of data to acoustic features. This results in mapping strategies that
seeks to communicate data to a listener by means of acoustic symbols that are seen to be made
meaningful through the application of some internal set of syntactical rules in the mind
(cognitive grammars). Ryle (1949) and Searle (1980) and Harnad (1991) have variously shown
that meaning cannot be generated for a listener in this way because, as Dreyfuss (1965) and
Polyani (1966) point out, objects of meaning require a background context against which their
meaning can be assigned, and the act of processing information provides no such context.
However, contemporary sonification research and practices seek to solve the mapping problem
whilst preserving the structure of the data during encoding so that it can be accurately
represented to a listener. Much contemporary sonification takes place in a context which
recognises ecological perspectives on the problem; that the very act of encoding data into sound
may also allow for aspects of its structure to be revealed via a listener’s engagement (Walker &
Kramer, 2004). Ecological, in this sense, situates the problems of perception within its inter-
dependent relationship with the sensory environment. In this context, the perceptualization
(Hermann, Hunt, Neuhoff, 2011) of data is important because different sensory modalities may
be useful for revealing different aspects of data structures. The heuristic processes of parsing
environmental sound and music (Bregman, 1990) may therefore be particularly helpful in a
search for meaningful patterns within data sets.
From this contemporary perspective, sonification may be seen as a structural investigation at
both the encoding and decoding stages. Whilst much attention has previously been focused on
strategies for encoding (also referred to here as mapping), we believe that considering these
alongside frameworks for decoding, based on contemporary theories of embodied auditory
perception and cognition may be of great significance for improving the efficacy of sonification.
While ecological approaches to sonification have been explored by a number of researchers in
the field, the current paper is concerned with approaches informed by embodied cognitive
science and musical aesthetics.
An additional problem with formalist auditory display/sonification approach is its treatment of
complexity. The discretized treatment of these materials also conforms to a computational–
formalist information processing paradigm (the Shannon–Weaver model of communication
(Shannon & Weaver, 1949) which considers (discrete) channels/dimensions with particular
Roddy and Bridges, Sounding Human with Data, MusTWork 2016 68
bandwidths, influenced by external sources of noise (via signal interference, or, from some
broader perspectives, a lack of contextualization allowing reconstruction of an ambiguous
communication). A formalist mapping approach based on the assumption of discrete dimensions
may be viewed as encompassing such a set of discrete channels, with the idea of additional
noise/ambiguity which that entails. An early commentary on the channel capacity/mapping
problem is to be found in George Miller’s12 commentary on human information–processing and
recognition abilities via working memory, ‘The Magical Number Seven, Plus or Minus Two’
(Miller, 1956). Miller’s work was informed by an analysis of Pollack’s (1952) early auditory
display studies which showed a communicative potential of c. 2.5 bits (6 elements/values
channel capacity) for unidimensional, i.e. single-modality, stimuli. Miller analyses the
experimental results of various contemporaries who were investigating auditory perception as an
information processing task to conclude that, for unidimensional judgments, working memory,
the kind of memory responsible for the short-term holding and processing of information, has a
capacity of 7 ± 2 chunks, or discrete objects, of information. He argues that a listener can
identify and remember roughly 7 ± 2 distinct pitches, amplitudes, rates of interruption, on-time
fraction, durations, and spatial locations when presented as domains for representing data. Miller
also noted that information capacity could be increased through additional parsing known as
chunking; items to be remembered could be associated with one another, freeing capacity in
formal, working memory.
This case is reserved for high–level cognition of precise rank-ordering relationships, etc.
Although it may be tempting to take this as a hard–and–fast rule (within its own somewhat
imprecise boundaries), it should be noted that Miller states clearly that this only holds true for
unidimensional stimulus cases and cannot describe the parsing of multidimensional stimuli e.g.
facial recognition, or, indeed, more complex musical cases. For example, even in more
‘unnatural’ formalist multidimensional cases, such as integrated frequency and amplitude
displays (Pollack, 1953), Miller comments that capacity has increased beyond the 7+-/-2, albeit
not in an obvious pattern noting the complexity of music13. Our own perspective is that such a
comparably unpredictable increase in capacity is due to ecological concerns.
Although the psychoacoustic paradigm expanded to encompass ecological psychoacoustics in
auditory display research (Walker and Kramer, 2004), the methods employed in this framework
draw heavily from the operationalist framework developed by Stanley S. Stevens, the founder of
psychoacoustics. Operationalism is a form of positivism which holds that a concept that cannot
be reduced to a measurement is meaningless. Stevens developed his cross-modal matching and
magnitude estimation techniques in order to reduce psychophysical information, e.g. heard
sounds, to simple measurements. However as Dreyfus (1965), Searle (1980), Johnson (1987) and
Polyani (1966) point out a large spectrum of human experience and human knowledge cannot be
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
";!V1)W=!et#al.!X;T"TY!+:7>(.!$#+$!01+:$40>,+$416Z!$#(!9>.467!$17($#(:!19!.2+,,(:!>64$.!46$1!,+:7(:!<#1,(.Z!4.!0:4$40+,!$1!
S4,,(:[.!$#(1:=!19!4691:2+$416!0#>6-467/!\#4,(!$#(!24,,(:!,424$!247#$!)(946(!$#(!2(0#+640+,!,424$.!19!%(:0(%$416Z!+$!$#(!
,(3(,!19!2>.40+,'.1640!(*%(:4(60(!$#(!4691:2+$416!46!$#(!0#>6-.!4.!018+:$40>,+$()!.1!$#+$!$#(!,4.$(6(:!0+6!!
(*%(:4(60(!:40#!2>.40+,!42+7(:=!+6)!7(.$>:(./!
"G!O1,,+0-!X"IKJY!.+<!H!)499(:(6$!+01>.$40!3+:4+?,(.!19!K!)499(:(6$!3+,>(.!(+0#Z!=4(,)467!Q/;!?4$.!X0/"KT!3+,>(.YZ!+!3+,>(!
<#40#!2+=!?(!.((6!0,1.(:!$1!:(+,8<1:,)!0+.(.!19!01764$43(]%(:0(%$>+,!:(01764$416!+?4,4$4(.!+6)!0,1.(:!$1!<#+$!247#$!?(!
34(<()!+.!:(+.16+?,(!^0#+66(,!0+%+04$4(._!91:!2>.40!X61$(!#1<!0,1.(!$#4.!947>:(!4.!$1!$#(!Q8?4$!:+67(!19!$#(!S`a`!XS>.40+,!
`6.$:>2(6$!a474$+,!`6$(:9+0(Y!.$+6)+:)b!!
Roddy and Bridges, Sounding Human with Data, MusTWork 2016 69
reduced to simple measurements, an issue which Miller’s (1956) account of memory and
information processing grapples with even as it seeks ways to quantify its capacity.
It is in these contexts that we argue that one crucial avenue to consider in the development of
sonification is an ecologically–considered model of auditory dimensions which is meaningful to
a listener as it aligns with the perception and interpretation strategies which we use within our
everyday environmental experience.
For more richly perceptualised sonification, as opposed to the narrower discretised signals of
auditory display, we are not concerned with chunking, per se, but with pattern and correlation
recognition, an entirely different problem within a perceptual, rather than formalist, context.
Leveraging our (integrative) cognitive–perceptual parsing systems may help us identify
meaningful patterns within multiparametric data (if degree of change is relatively constrained). It
is in this context that we contend that an ecological perspective on the mapping problem is
crucial in certain contexts: those where the data is mapped for clear communication; for
example, for didactic purposes. In this context, our interpretative framing of multiparametric data
may be aided by a consideration of models (schemas) derived from common perceptual
experience cases. These may, in part, be explained as the ‘environmental regularities’ which
underpin heuristic principles within our perceptual processes. But these regularities are more
basic structural framing principles rather than necessarily supporting interpretative framing. To
consider how interpretative framing beyond the basics happens, we may need to consider how
our experience of the environment impacts upon our conceptual systems.
Sense-making in Sound from Perception to Cognition
To restate our perspective on sonification, it is our contention that sonification is not just
rendering, but also the act of framing data as it is ‘filtered’ through our perceptual transduction
and expectancy schemas; our ‘sense-making’ apparatus. This perceptualization (Hermann, Hunt,
Neuhoff, 2011) is thus more than just a mapping from one domain to another, but also entails an
act of structural framing which derives from our perceptual systems. As such, approaching
sonification simply from the perspective of formalism as opposed to being perceptually and
cognitively informed may impede its ability to meaningfully represent data (Worrall has
compared with serial music ‘problem’, see also (Lerdahl, 1988, McAdams, 1987).
Perceptualization involves not only leveraging familiar environmental contexts but also
leveraging from embodied contexts in ways that are compatible with the faculties and processes
of embodied cognition. Patterns within complex data may be revealed by rich mappings which
are engaging enough to support careful listening and communicative enough to furnish the
listener with the required information. But, beyond even these concerns, there is the question of
how conceptual framings arise out of the basic perceptual–structural framings through which we
approach sonification. Theories of embodied cognition place the emergent patterns of experience
that arise in the interaction between structural regularities of the environment and the structural
regularities of the human body at the center of the ‘problem’ of conceptualisation.
Electroacoustic Music Theory, Embodied Cognition and Sonification
Embodied cognitive science examines the relationship between the body and mind with a
specific focus on how the physical and perceptual dimensions of the human body shape and
define the cognitive processes and conceptual systems of the human mind. It emerged in the late
20th century as researchers began to study emotion, culture and aesthetic experience. It has
shaped the development of a number of important disciplines related to sonification research and
practice, e.g. computer science, artificial intelligence and human computer interaction (Brooks,
2003; Dourish, 2004; Imaz and Benyon, 2007), computer music (Leman, 2008; Klemmer et al.,
Roddy and Bridges, Sounding Human with Data, MusTWork 2016 70
2006), cognitive sciences (Varela et al., 1991), visual perception (Noë, 2009), aesthetics
(Johnson, 2013), music (Godøy, 2006;, 2005; Brower, 2000; Larson, 2010; Cox, 2001),
linguistics and philosophy (Lakoff and Johnson, 1999).
Embodied cognition researchers have presented a number of theoretical models which describe
how the embodied mind perceives meaning in and applies meaning to it’s world. Image schemas
were first introduced by Lakoff and Johnson (1987) and can be thought of as ‘gestures of
thought’ in that they are basic gestural patterns derived from sensorimotor experience which we
draw upon to structure our thinking and conceptual systems. The process by which these basic
patterns of experience are imported into cognition is referred to as conceptual metaphor (Lakoff
and Johnson 1980). The process by which multiple mental spaces are integrated to create new
mental content s referred to as conceptual blending (Fauconnier and Turner 2002). A number of
researchers have described musical listening and composition in terms of embodied schemata,
conceptual metaphors and conceptual blends: Kendall (2014), Cox (2000), Brower (2000);
Adlington (2003) Godøy (2003; 2006) Wilkie et. al. (2010). These thinkers make the argument
that the embodied components of cognition represented in these theoretical models play a key
role in the listener’s experience of music. In electroacoustic music theory, Spectromorphology is
a descriptive framework for electroacoustic music consisting of detailed categorisation schemes
deriving from basic gestural shapes called primal gestures that are extended to add a meaningful
low-level organisational structure to musical domains (Smalley, 1997).
Sound(ing) Schemas and Embodied Functions in Electronic/Electroacoustic
Music
Previous work by one of the authors (Graham and Bridges 2014a; 2014b) has investigated points
of compatibility between Smalley’s spectromorphology and the embodied image schema theory
of Lakoff and Johnson. They argue (ibid.) that Smalley’s gestural surrogacy and the dimensions
of his gestures are compatible with image schema theory and its extension by Johnson (2007) in
terms of qualitative dimensions of movement (essentially, gestural details of more regular or
chaotic behaviour which alter some of the contours within image schemas).
Particular points of comparison between Smalley (1997) and Lakoff and Johnson’s work,
especially its extension by Johnson (2007):
General: environmental models/embodied–ecological models of causality, ideas of
musical ‘forces’ based on environmental analogues
General: idea of ‘embodied functional associations’ of particular movements
Specific: image schema structures (cycles, verticality, source–path–goal, container)
identifiable within spectromorphologies
Specific: dimensions of Smalley’s sound gestures are similar to Johnson’s qualitative
dimensions of movement
The similarity between the dimensions of Smalley’s sound gestures (termed energy–motion
profiles) and Johnson’s qualitative dimensions of movement can be seen below (Bridges and
Graham, 2014a).
Roddy and Bridges, Sounding Human with Data, MusTWork 2016 71
Johnson (2007)
Embodied Association
Tension
Rate–effort=>overcoming
inertia
Projection
Sudden rate-change/ transient
movement
Linearity
Coherence of path
Table 1.1 Embodiment and Spectromorphology (Bridges and Graham, 2014a)
Not only are these dimensions of movement similar in terms of the division of embodied
associations, but they also relate closely to the basic schematic forms of verticality and source–
path goal. Smalley’s logic of environmental causality sees certain sound gestures as embodying
more rootedness or dynamism (projection/motion launching). We believe that these ideas of
certain timbres as providing temporal structural dynamics may help us to develop more
convincing ‘sonic narratives’ using data if theories such as spectromorphology and exploratory
practices within electronic music can help to move us in the direction of embodied theories of
musical timbre.
This emphasis on physicality within novel musical structures is also to be found beyond the
world of electronic and electroacoustic music. Whilst common practice tonal music could be said
to be based on structures explicable via the metaphor Music–as–Moving–Force (Johnson, 2007),
Adlington (2003) explores image schema theory and contemporary music from the perspective
that salient metaphors may relate more to ideas of changes of material and changes of physical
state. The key developmental aspect which this highlights for our present purposes is that sonic
‘image’ schemas may be best viewed as temporally dynamic and morphologically/structurally
plastic.
There is much still to explore in terms of how the specific domains of sound and music can be
addressed via image schema theory. The common auditory–perceptual affordances of stream
segregation and integration (Bregman, 1990) and the material metaphors of glitch/rupture,
stretching and bouncing/inertial effects which are observable in a variety of contemporary
electronic musical processes have the potential to be useful in sonification mappings (indeed,
where these configurations occur unintentionally within existing sonifications, they may already
act as clues to significant elements within the data). Sound’s perceptual–ecological interpretative
frames (contextual framing) occurs within commonplace perceptual experience due to the
alignment of perceptual–heuristic processes with ‘environmental regularities’ (Bregman, 1993).
Combining these inbuilt dynamics with more attention to potential embodied timbral/textural
mappings could lead to a much more sophisticated integrating approach which avoids the
obscuring of meaningful sonic dimensions behind inappropriate formal models. Exploratory
creative processes which investigate embodied mapping strategies may help to suggest further
avenues for the development of accessible sonic mappings.
A Consideration of The Human Cost: A Data-driven Composition Using
Embodied Sonifiction Techniques
The Human Cost is a piece of data driven music composed by one of the authors (see Roddy,
2015), in which principles from embodied cognitive science are applied to organise mapping
strategies from data to sound in a sonification context. Some of the embodied dimensions of this
Roddy and Bridges, Sounding Human with Data, MusTWork 2016 72
piece are considered in this section. The piece is intended to communicate a sense of the human
cost of Ireland’s economic recession.
The Human Cost was motivated by Smalley’s statement that “in electroacoustic music the voice
always announces a human presence” (Smalley, 1996). It was thought that as result the human
voice might prove effective for representing data which measured the lives and behaviors of
people. As such it was decided to sonify socioeconomic data sets from the period of Ireland’s
economic crash and recession. Deprivation, unemployment, emigration and GNP rates in Ireland
from 2007 to 2012 were chosen as data sets for sonification. Rich multi-layered mapping
strategies were employed to sonify this data. GNP is mapped to control a parameterised sounding
object intending to reflect the sound of a heartbeat as GNP falls the heartbeat slows and as it
rises the heartbeat increases. The choice of the heartbeat sound to represent GNP data was
informed by White (2003) who argues that the economy is often conceptualised as a living
organism. Deprivation, unemployment and emigration are mapped to control the prosodic
features of three synthesised vocal simulations. The simulation to which the emigration rate was
mapped acts as a “lead line” and the pitch and prosodic content of the vocal gesture are
modulated to imitate the kinds of structure found in the old Irish laments, a type of song sang at a
wake, a kind of funeral celebration which was often held to honour either a deceased relative, or
a relative who was emigrating with no prospect of return.
Laments represent a cultural connection with the historical (and contemporary) experience of the
emigration of the Irish Diaspora, cultural forms in which the singer’s personal experiences of the
passing or emigration of a loved one are expressed and communicated through vocal gesture.
This transduction of human experience to physical, sound-producing gestures represented a
useful physical-emotive mapping of relevance to the data sonified. The data is mapped so that
the lead voice takes the foreground while the other two voices present a form of backing and the
heartbeat performs a grounding percussive role in the piece. Deprivation rate and unemployment
are mapped to these backing voices. Data is mapped to control the vowel shape in each vocal
simulation so that as the economy worsens the open vowel sounds shift to closed vowel sounds
to communicate a sense of tension. It is also mapped to spatial parameters so that both vocal
simulations move through space and as the data increases and decreases the speed at which they
move through space also increases and decreases creating a sense of frenzy in the piece as the
economy crashes!
Conclusion
There is more to sound and music than pitches, durations, timbres and amplitudes. Sound is a
powerful medium for the representation of data precisely because of its communicative
dimensions; some of which are unaccounted for in standard sonic models based on discrete
dimensions and parameterisation.
We have argued that a new conceptual model and specification of sound which recognises the
embodied and aesthetic dimensions of sound is crucial to the development of effective data to
sound mapping strategies. If sonification involves the mapping of data to sound for the purposes
of communicating information about a data-source, this necessarily re-frames the information in
the data in terms of the embodied and aesthetic and dimensions (and dimensional integrating
effects) of the chosen sound materials. Such a process of reframing has the potential to integrate
insights from a diverse range of sonic practices and theories, from embodied cognitive science
and ecological psychology to electronic/electroacoustic music composition and production.
Roddy and Bridges, Sounding Human with Data, MusTWork 2016 73
Framing of this nature can be used to ensure that sonification mapping strategies are a good fit
for the listener’s cognitive meaning-making faculties, thus supporting the efficient
communication of the data. They can also be used to explore emotional and affective dimensions
to a sonification, thus presenting a richer representation of the data than would otherwise be
possible. The development of sonification within this context is best seen as an integration of the
arts and sciences as their interests intersect within the spheres of sound, perception and meaning–
making.
Acknowledgements
This publication has emanated from research supported in part by a research grant from Science
Foundation Ireland (SFI) and is co-funded under the European Regional Development Fund
under Grant Number 13/RC/2077.
References
Adlington, R. (2003). Moving beyond motion: Metaphors for changing sound. Journal of the Royal Musical
Association, 128(2), 297-318.
Ballora, M., Pennycook, B., Ivanov, P. C., Glass, L., & Goldberger, A. L. (2004). Heart rate sonification: A new
approach to medical diagnosis. Leonardo, 37(1), 41-46.
Barrass, S. (2012). The aesthetic turn in sonification towards a social and cultural medium. AI & society, 27(2), 177-
181.
Barrass, S., & Barrass, T. (2013). Embedding sonifications in things.
Barrett, N., & Mair, K. (2014, May). Sonification for geoscience: Listening to faults from the inside. In EGU
General Assembly Conference Abstracts (Vol. 16, p. 4489).
Bregman, A. S. (1990). Auditory scene analysis: The perceptual organization of sound. MIT press.
Bregman, A. S. (1993). Auditory scene analysis: Hearing in complex environments. Thinking in sound: The
cognitive psychology of human audition, 10-36.
Brewster, S. A. (1994). Providing a structured method for integrating non-speech audio into human-computer
interfaces (Doctoral dissertation, University of York).
Brooks, R. A. (1991). Intelligence without representation. Artificial intelligence, 47(1), 139-159.
Brower, C. (2000). A cognitive theory of musical meaning. Journal of music theory, 44(2), 323-379.
Clark, A. (2001). Natural-born Cyborgs. New York: OUP USA .
Cox, A. (2001). The mimetic hypothesis and embodied musical meaning. Musicae Scientiae, 5(2), 195-212.
Dourish, P. (2004). Where the action is: the foundations of embodied interaction. MIT press.
Dreyfus, H. L. (1965). Alchemy and artificial intelligence. Santa Monica, CA: The Rand Corporation, Research
Report.
Fairfax, T., Laing, C., & Vickers, P. (2014). Network Situational Awareness: Sonification and Visualization.
Handbook of Research on Digital Crime, Cyberspace Security, and Information Assurance, 334.
Fitch, W. T., & Kramer, G. (1994). Sonifying the body electric: Superiority of an auditory over a visual display in a
complex, multivariate system. In Santa Fe Institute Studies in the Science of Complexity Proceedings (Vol. 18,
pp. 307-307). Addison-Wesley Publishing Co.
Flowers, J. H. (2005). Thirteen years of reflection on auditory graphing: Promises, pitfalls, and potential new
directions. Faculty Publications, Department of Psychology, 430.
Godøy, R. I. (2003). Motor-mimetic music cognition. Leonardo, 36(4), 317-319.
Godøy, R. I. (2006). Gestural-Sonorous Objects: embodied extensions of Schaeffer's conceptual apparatus.
Organised Sound, 11(02), 149-157.
Godøy, R. I., Jensenius, A. R., & Nymoen, K. (2010). Chunking in music by coarticulation. Acta Acustica united
with Acustica, 96(4), 690-700.
Graham, R., & Bridges, B. (2014a). Gesture and Embodied Metaphor in Spatial Music Performance Systems
Design. In NIME (pp. 581-584).
Graham, R., & Bridges, B. (2014b). Strategies for spatial music performance: the practicalities and aesthetics of
responsive systems design. Divergence Press, 3, December 2014. University of Huddersfield Press/Centre for
Research in New Music (CeReNeM). [Online journal article]. Available at: DOI: 10.5920/divp.2015.33. [last
accessed 22 July 2016].
Grond, F. & Berger, J. (2011). Parameter mapping sonification. In Hermann, T., Hunt, A., Neuhoff, J. G., (Eds) The
Sonification Handbook, (pp. 363397). Berlin, Germany: Logos Publishing House.
Harnad, S. (1990). The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1), 335-346.
Hermann, T., & Hunt, A. (2005). Guest editors' introduction: An introduction to interactive sonification. IEEE
Roddy and Bridges, Sounding Human with Data, MusTWork 2016 74
multimedia, (2), 20-24.
Hermann, T., Hunt, A. & Neuhoff, J. G. (2011). Introduction. In Hermann, T., Hunt, A., Neuhoff, J. G., editors, The
Sonification Handbook, (pp.16) Berlin, Germany; Logos Publishing House.
Hermann, T., & Ritter, H. (1999). Listen to your data: Model-based sonification for data analysis. Advances in
intelligent computing and multimedia systems, 8, 189-194.
Huron, D. B. (2006). Sweet anticipation: Music and the psychology of expectation. MIT press.
Imaz, M., & Benyon, D. (2007). Designing with blends: Conceptual foundations of human-computer interaction and
software engineering methods. Mit Press.
Isaacson, R. E., & Aust, R. J. (2007, June). The Use of Sonification in Synthetic Voice Messages. In EdMedia:
World Conference on Educational Media and Technology (Vol. 2007, No. 1, pp. 957-965).
Johnson, M. (1987). The body in the mind: The bodily basis of meaning, imagination, and reason. University of
Chicago Press.
Johnson, M. (2013). II. Identity, Bodily Meaning, and Art. Art and Identity: Essays on the Aesthetic Creation of
Mind, 32, 15.
Kendall, G. S. (2014). The Feeling Blend: Feeling and emotion in electroacoustic art. Organised Sound, 19(2), 192.
Klemmer, S. R., Hartmann, B., & Takayama, L. (2006, June). How bodies matter: five themes for interaction
design. In Proceedings of the 6th conference on Designing Interactive systems (pp. 140-149). ACM.
Larson, S. (2012). Musical forces: Motion, metaphor, and meaning in music. Indiana University Press.
Lakoff, G.. Women, Fire and Dangerous Things. What Categories Reveal about the Mind. University of Chicago
Press.
Lakoff, G., & Johnson, M. (1980). Metaphors we live by. University of Chicago Press.
Lakoff, G., & Johnson, M. (1999). Philosophy in the flesh: The embodied mind and its challenge to western thought.
Basic books.
Leman, M. (2008). Embodied music cognition and mediation technology. Mit Press.
Lerdahl, F. (1988). Tonal pitch space. Music Perception: An Interdisciplinary Journal, 5(3), 315-349.
Lockton, D., Bowden, F., Brass, C., & Gheerawo, R. (2014). Powerchord: Towards ambient appliance-level
electricity use feedback through real-time sonification. In Ubiquitous Computing and Ambient Intelligence.
Personalisation and User Adapted Services (pp. 48-51). Springer International Publishing.
McAdams, S. (1987). Music: A science of the mind?. Contemporary Music Review, 2(1), 1-61.
Miller, G. (1956). The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing
Information. The Psychological Review, vol. 63, pp. 81-97.
Noë, A. (2009). Out of our heads: Why you are not your brain, and other lessons from the biology of consciousness.
Macmillan.
Park, Tae Hong, Jonathan Turner, Christopher Jacoby, Alex Marse, Michael Musick, Ajay Kapur, and Jinyin He.
Locative sonification: Playing the world through citygram. Ann Arbor, MI: Michigan Publishing, University of
Michigan Library, 2013.
Peres, S. C. (2012). A comparison of sound dimensions for auditory graphs: Pitch is not so perfect. Journal of the
Audio Engineering Society, 60(7/8), 561-567.
Peres, S.C., & Lane, D.M. (2005). Auditory graphs: The effects of redundant dimensions and divided attention.
Proceedings of the 11th International Conference on Auditory Display (ICAD2005) (pp.169-174), Limerick,
Ireland.
Polli, A. (2012). Soundscape, sonification, and sound activism. AI & society, 27(2), 257-268.
Pollack, I. The assimilation of sequentially encoded information. Amer. J. Psychol., 1953, 66, 421-435.
Pollack, I. The information of elementary auditory displays. J. Acoust. Soc. Amer., 1952, 24, 745-749.
Pollack, I. The information of elementary auditory displays. II. J. Acoust. Soc. Amer., 1953, 25, 765-769.
Polanyi, M. (1966). The Tacit Dimension. Anchor. Garden City, NY.
Rimland, J., Ballora, M., & Shumaker, W. (2013, May). Beyond visualization of big data: a multi-stage data
exploration approach using visualization, sonification, and storification. In SPIE Defense, Security, and Sensing
(pp. 87580K-87580K). International Society for Optics and Photonics.
Roddy, S. (2015). Embodied Sonification. Doctoral Dissertation. Trinity College Dublin. Ireland.
Roddy, S., & Furlong, D. (2014). Embodied aesthetics in auditory display. Organised Sound, 19(01), 70-77.
Roddy, S., & Furlong, D. (2015, July). Sonification listening: An empirical embodied approach. In Proceedings of
the 21st International Conference on Auditory Display (ICAD 2015).
Ryle, G. (1949). The concept of mind. London, UK: Hutchinson.
Searle, J. R. (1980). Minds, brains, and programs. Behavioral and Brain Sciences, 3(03), 417-424.
Shannon, C. E., & Weaver, W. (1949). The mathematical theory of information. Urbana, Illinois: University of
Illinois Press.
Roddy and Bridges, Sounding Human with Data, MusTWork 2016 75
Sjuve, E. (2015, July). Metopia: experiencing complex environmental data through sound. In Proceedings of the
21st International Conference on Auditory Display (ICAD 2015).
Smalley, D. (1996). The listening imagination: listening in the electroacoustic era. Contemporary Music Review,
13(2), 77-107.
Smalley, D. (1997). Spectromorphology: explaining sound-shapes. Organised sound, 2(02), 107-126.
Stevens, S. S. (1975). Psychophysics. Transaction Publishers.
Supper, A. (2014). Sublime frequencies: The construction of sublime listening experiences in the sonification of
scientific data. Social Studies of Science, 44(1), 34-58.
Varela, F. J., & Thompson, E. &E. Rosch. (1991). The Embodied Mind: cognitive science and human experience.
Cambridge, Mass.: MIT Press.
Walker, B. N. (2000). Magnitude estimation of conceptual data dimensions for use in sonification. Doctoral
Dissertation. Rice University, Houston, TX.
Walker, B. N., & Kramer, G. (2004). Ecological psychoacoustics and auditory displays: Hearing, grouping, and
meaning making. Ecological psychoacoustics, 150-175.
White, M. (2003). Metaphor and economics: the case of growth. English for specific purposes, 22(2), 131-151.
Wolf, K., & Fiebrink, R. (2013). SonNet: A Code Interface for Sonifying Computer Network Data. In NIME
(pp. 503-506).
Worrall, D. (2009). Sonification and Information: Concepts, instruments and techniques. PhD thesis, University of
Canberra.
Worrall, D. (2010). Parameter mapping sonic articulation and the perceiving body.
Wilkie, K., Holland, S., & Mulholland, P. (2010). What can the language of musicians tell us about music
interaction design?. Computer Music Journal, 34(4), 34-48.
Yoshida, T., Kitani, K. M., Koike, H., Belongie, S., & Schlei, K. (2011, March). EdgeSonic: image feature
sonification for the visually impaired. In Proceedings of the 2nd Augmented Human International Conference (p.
11). ACM.
Zbikowski, L. M. (2002). Conceptualizing music: Cognitive structure, theory, and analysis. Oxford University
Press.
Roddy and Bridges, Sounding Human with Data, MusTWork 2016 76
... Similar concerns have been raised within sound studies and practices, notably Truax [4], who criticised the overreliance on the 'energy transfer model' of sound (asserting that a psychophysical approach does not account for many aspects of sound's communicative afordances), O'Callaghan [5], a philosopher of sound, and sound artists and sound studies theorists Kahn [6], LaBelle [7] and Cox [8]. The relationship between arts practices and the soniication mapping problem is further discussed by Roddy and Bridges [9,10]. From a design-centered perspective Worrall [3,11,12] presents a similar argument: that the software tools used in soniication parameterise sound using the basic parameters of Western tonal music (pitch, duration, loudness and timbral identity/ diference), an example being the PMSon mapping of pitch, loudness, duration and timbre to unique data [13]). ...
... Conceptual blending is another process by which familiar conceptual content is integrated to generate new hybrid conceptual content [22]. Conceptual blending and its relationship to sonic information design is explored elsewhere [9], and design approaches informed by embodied cognition have been successfully applied in the context of HCI [23][24][25][26]. In a similar fashion, embodied approaches to interactive sonic information design have been developed, informed by Dourish's [27] introduction of the concept of embodied interaction; see (Serain et al. 2011). ...
Article
Full-text available
This is a theoretical paper that considers the mapping problem, a foundational issue which arises when designing a sonification, as it applies to sonic information design. We argue that this problem can be addressed by using models from the field of embodied cognitive science, including embodied image schema theory, conceptual metaphor theory and conceptual blends, and from research which treats sound and musical structures using these models, when mapping data to sound. However, there are currently very few theoretical frameworks for applying embodied cognition principles in a sonic information design context. This article describes one such framework, the embodied sonification listening model, which provides a theoretical description of sonification listening in terms of conceptual metaphor theory.
... Miranda's [35] Evolutionary Computer Music, comprised of chapters authored by key thinkers and practitioners in the field, has contributed to the systematization of thinking in regards to evolutionary methods in generative music while Boden's [36][37] research on the relationship between creativity and computing has influenced the design of a number of generative music systems [ [43]. While advances have been taking place in the field of generative music much greater advances have been taking place in computing more generally with the emergence and maturation of the Internet and recent developments in IoT, cloud, mobile and smart technologies [44]. These technologies have reshaped Western culture and defined a new set of practices around how we consume and interpret information and create and listen to music [13] [44]. ...
... While advances have been taking place in the field of generative music much greater advances have been taking place in computing more generally with the emergence and maturation of the Internet and recent developments in IoT, cloud, mobile and smart technologies [44]. These technologies have reshaped Western culture and defined a new set of practices around how we consume and interpret information and create and listen to music [13] [44]. This cultural shift towards an online and networked approach to information consumption and music listening needs to be addressed and accounted for in the design of a system which is intended to present key information to a listener through music. ...
Conference Paper
Full-text available
This is a position paper which describes work in progress to develop an AI/ML driven auditory ambient information system which incorporates generative music techniques and considers some of the factors involved the design and development of a system of this nature. The system is intended to represent and communicate information about cryptocurrency markets to a user. The generative music system uses evolutionary computing and machine learning techniques, which are driven by the original cryptocurrency data, to create musical information. This information is then mapped to synthesis parameters. The implementation described is web based running client-side in the browser. The paper describes the relevant background literature before presenting a generalised model for a system of this nature before discussing aspects of the implementation in its current state. The paper closes with a discussion of the system and future work.
... This embraces the notion of sequential auditory perception loops as Interactive Sonifcations [12]; however, it stands tendentially within the purpose of data analysis and a somewhat rigid perspective of sound perception as function. Multiple studies have since been developed that seek alternative views and explorations within the feld, grasping embodied metaphors and the role of aesthetics for sonifcation design [13,14,15,16]. ...
Conference Paper
Full-text available
The act of sound perception and its subjective dimensions, from physical to psychoacoustics, from semantic to affective, carry an inherent challenge for the conception and evaluation of every audio-based artefact. Starting from a previous framework of evaluation approaches, we seek to deconstruct the confguring elements of these processes, searching for theoretical foundations informing Sound Design and possible applications for Auditory Displays. This work is a frst step into identifying a body of knowledge on the listener’s experience, how the act of listening takes place and how the sequence of listening actions can evolve as forms of dialogue, creating dialogical spaces for making sense of auditory information. With this work, practitioners can gain new insights into how existing techniques for creating auditory artefacts can be confgured and transformed into new, alternative approaches.
... • Data-to-sound mapping reflected an approach based on the use of conceptual metaphors to map data dimensions into sound dimensions. Conceptual metaphors, a concept introduced in 1980 by George Lakoff and Mark Johnson [41] are not new to the world of data sonification [12][13] [42] [43]. In this case, we designed the mapping strategy based on the metaphor that through distorted sound, a listener would gauge distortion in the data represented by the specific sound i.e., that hearing something wrong in the sound would metaphorically communicate the expectation of something wrong in the data. ...
Article
Full-text available
INTRODUCTION: We present early results of an ongoing investigation into the use of sound for real-time monitoring of anomalous behaviour in digital and digital/physical systems. OBJECTIVES: We aim to define design guidelines to both support authors in the process of creating sonifications that are both efficient and engaging and the transition of sonification into a mass medium for the representation of data in everyday life. METHODS: Through two Design Actions, we apply Design Research to the definition of the use case, the interaction paradigm and the experimental protocol for real-world evaluation of sonification tools. RESULTS: Two Design Actions are described. Methodologies and results of the first experimental phase are presented in detail along with their influence on the second phase, currently ongoing. CONCLUSION: We sketch a tentative design-driven process for sonifications for the real-time monitoring of anomalous behaviour in digital and digital/physical systems.
Article
The aim of this study is to integrate the aesthetic theory into an immersive data imaging exhibition that shows how aesthetic principles influence user engagement, visual appealing and complex interaction in Tri Sphere Technologies. The study used mixed methodology for evaluating immersive visualization styles on interaction complexity, typography, average rating, color harmony and information architecture as well as aesthetics. The qualitative analysis investigated the immersive user engagement, exploration of data and aesthetic appealing in Tri Sphere Technologies visualizations. The data analysis revealed the complicated patterns and connections among aesthetic elements and modalities in immersive visualization like how Tri Sphere Technologies execute the thematic analysis in this context. This research provides the practical and theoretical implications of data exhibition design in the context of aesthetic theory. The results show how theory affects user experience in difficult varied immersive situations and also highlight the complex connections among principles and user preferences in aesthetic theory. While selecting immersive visualization forms, this study focused on the relevancy of interactivity, aesthetics in visualization and the nature of data and information. This research influences data visualization, Human-Computer Interaction and technological design in an immersive interaction context. For further research, the research also discussed the research gaps by linking the cultural aesthetics preferences and data visualization long term impact on engagement and user learning in the context of immersive visualization. The research advanced the knowledge and techniques for constructing data exhibition and beautiful effective visualization.
Article
Full-text available
The aesthetic dimension has been proposed as a potential expansion of sonification design, creating listening pieces that reach the goal of effective data communication. However, current views of aesthetics still aim at optimising mapping criteria to convey the ‘right meaning’, maintaining a mostly functional view on what is considered a successful sonification. This paper proposes an interdisciplinary approach to the aesthetics of sonification experience, grounded on theoretical foundations from phenomenology of interaction, post-phenomenology, cross-cultural studies, acoustic ecology and deep listening. From this journey, we present the following design insights: (1) the design of sonifications becomes a design for experience, (2) co-designed during the interaction with each participant; (3) the sonification artefact gains a mediating role that guides the participant’s intentions in the sonification space; (4) the aesthetics of a sonification experience generates a multistable phenomenon, offering new opportunities to experience multiple perspectives over data; (5) the interaction between human participants and the sonic emanations compose a dialogic space. A call for action to reframe the sonification field into novel design spaces is now open, with aesthetics gaining a transformational role in sonification design and interaction.
Article
Full-text available
Introducción: el auge de la ficción sonora actual exige de los estudios en Comunicación una atención que compruebe las capacidades de los estudiantes en creación sonora ficcional para su futuro laboral. La dependencia de la imagen visual y la palabra que las jóvenes generaciones poseen exige, a su vez, una comprobación empírica. Metodología: la experiencia que ahora se presenta lleva a cabo una acción de escritura de microrrelatos de ficción sonora, diseñada para detectar las dificultades encontradas por las y los estudiantes al poner en práctica todos los elementos sonoros con ausencia del de palabra, para luego incorporarla progresivamente en distintas fases de escritura de guion. El trabajo de escritura guiada se completa con un doble movimiento: uno analítico sobre los guiones realizados y otro basado en las encuestas cumplimentadas por los autores y autoras sobre la complejidad de esta tarea de guionización. Resultados: los microrrelatos generados contaron con una carencia casi total de recursos sonoros no verbales, sin explotación de otros elementos del lenguaje sonoro. Discusión: los resultados confirman una “sordera funcional” que deriva en una falla para construir relatos sonoros de ficción, más allá del diseño de paisajes sonoros. Conclusiones: el análisis de las creaciones apunta a la existencia de una variación en la narrativa cognitiva de las nuevas y futuras generaciones de creadores y de oyentes en lo que a sonido respecta.
Article
We present a first-person, retrospective exploration of two radio sonification pieces that employ narrative scaffolding to teach audiences how to listen to data. To decelerate and articulate design processes that occurred at the rapid pace of radio production, the sound designer and producer wrote retrospective design accounts. We then revisited the radio pieces through principles drawn from guidance design, data storytelling, visualization literacy, and sound studies. Finally, we speculated how these principles might be applied through interactive, voice-based technologies. First-person methods enabled us to access the implicit knowledge embedded in radio production and translate it to technologies of interest to the human–computer-interaction community, such as voice user interfaces that rely on auditory display. Traditionally, sonification practitioners have focused more on generating sounds than on teaching people how to listen; our process, however, treated sound and narrative as a holistic, sonic-narrative experience. Our first-person retrospection illuminated the role of narrative in designing to support people as they learn to listen to data.
Article
Full-text available
Data sonification is a practice for conducting scientific analysis through the use of sound to represent data. It is now transitioning to a practice for communicating and reaching wider publics by expanding the range of languages and senses for understanding complexity in data-intensive societies. Communicating to wider publics, though, requires that authors intentionally shape sonification in ways that consider the goals and contexts in which publics relate. It requires a specific set of knowledge and skills that design as a discipline could provide. In this article, we interpret five recent sonification projects and locate them on a scale of intentionality in how authors communicate socially relevant issues to publics.
Conference Paper
Full-text available
This paper presents a sonification listening model built from models of embodied cognitive meaning-making faculties. The aim of such a model is to aid in understanding how meaning is applied to auditory stimuli at the cognitive level.this in turn can aid auditory display designers in creating more effective auditory displays. The concept of 'scale' in sonification is considered in relation to the faculties described in the model. An experiment that explores how embodied auditory cognition, as described by the model, understands and interprets sonifications is then presented. This examining two specific kinds of 'scale models' listeners employ to interpret a sonification. The results obtained from this experiment are particularly convincing showing that a listeners knowledge of the data-set being sonified will determine how they interpret changes in the auditory stimuli in a sonification. The existence of these scale models, the impact of a listeners knowledge on their perception of a sonification and the implications imposed by the embodied nature of auditory cognition suggest a new avenue for auditory display researchers interested in developing meaningful sonifications that exploit the embodied nature of auditory cognition.
Article
Full-text available
Starting from the assumption that meaning in electroacoustic music is an outcome of the listener's mental processes, it is the goal of this essay to explicate the mental processes whereby feeling and emotion contribute to meaning when listening to electroacoustic music. This essay begins with a broad consideration of feeling and emotion with an eye toward artistic experience, spanning from basic emotions to nuanced phenomenal qualities. It then introduces the concept of mental layers in support of the multi-levelled nature of meaning, especially in this case, meaning that is felt as well as comprehended. These two preliminary topics precede the introduction of the feeling blend, an extension of blend theory as presented by Fauconnier and Turner (2002). Core issues for blend theory, such as what constitutes a mental space and what triggers a blend, are reconsidered in the light of practical examples from the literature of electroacoustic music. In conclusion, the feeling blend is proposed as an essential concept to understanding artistic experience and an intrinsic aspect of being human.
Conference Paper
Full-text available
As the sheer volume of data grows exponentially, it becomes increasingly difficult for existing visualization techniques to keep pace. The sonification field attempts to address this issue by enlisting our auditory senses to detect anomalies or complex events that are difficult to detect via visualization alone. Storification attempts to improve analyst understanding by converting data streams into organized narratives describing the data at a higher level of abstraction than the input stream that they area derived from. While these techniques hold a great deal of promise, they also each have a unique set of challenges that must be overcome. Sonification techniques must represent a broad variety of distributed heterogeneous data and present it to the analyst/listener in a manner that doesn’t require extended listening – as visual “snapshots” are useful but auditory sounds only exist over time. Storification still faces many human-computer interface (HCI) challenges as well as technical hurdles related to automatically generating a logical narrative from lower-level data streams. This paper proposes a novel approach that utilizes a service oriented architecture (SOA)-based hybrid visualization/ sonification / storification framework to enable distributed human-in-the-loop processing of data in a manner that makes optimized usage of both visual and auditory processing pathways while also leveraging the value of narrative explication of data streams. It addresses the benefits and shortcomings of each processing modality and discusses information infrastructure and data representation concerns required with their utilization in a distributed environment. We present a generalizable approach with a broad range of applications including cyber security, medical informatics, facilitation of energy savings in “smart” buildings, and detection of natural and man-made disasters.
Article
This article can be viewed as an attempt to explore the consequences of two propositions. (I) Intentionality in human beings (and animals) is a product of causal features of the brain. I assume this is an empirical fact about the actual causal relations between mental processes and brains. It says simply that certain bran processes are sufficient for intentionality. (2) Instantiating a computer program is never by itself a sufficient condition of intentionality. The main argument of this paper is directed at establishing this claim. The form of the argument is to show how a human agent could instantiate the program and still not have the relevant intentionality. These two propositions have the following consequences: (3) The explanation of how the brain produces intentionality cannot be that it does it by instantiating a computer program. This is a strict logical consequence of 1 and 2. (4) Any mechanism capable of producing intentionality must have causal powers equal to those of the brain. This is meant to be a trivial consequence of 1. (5) Any attempt literally to create intentionality artificially (strong AI) could not succeed just by designing programs but would have to duplicate the causal powers of the human brain. This follows from 2 and 4. 'Could a machine think?' On the argument advanced here only a machine could think, and only very special kinds of machines, namely brains and machines with internal causal powers equivalent to those of brains. And that is why strong AI has little to tell us about thinking, since it is not about machines but about programs, and no program by itself is sufficient for thinking.