Musical sonification supports visual discrimination of color intensity

Abstract

Visual representations of data introduce several possible challenges for the human visual perception system in perceiving brightness levels. Overcoming these challenges might be simplified by adding sound to the representation. This is called sonification. As sonification provides additional information to the visual information, sonification could be useful in supporting the visual perception. In the present study, usefulness (in terms of accuracy and response time) of sonification was investigated with an interactive sonification test. In the test, participants were asked to identify the highest brightness level in a monochrome visual representation. The task was performed in four conditions, one with no sonification and three with different sonification settings. The results show that sonification is useful, as measured by higher task accuracy, and that the participant's musicality facilitates the use of sonification with better performance when sonification was used. The results were also supported by subjective measurements, where participants reported an experienced benefit of sonification.

Full text

Full Terms & Conditions of access and use can be found at
https://www.tandfonline.com/action/journalInformation?journalCode=tbit20
Behaviour & Information Technology
ISSN: 0144-929X (Print) 1362-3001 (Online) Journal homepage: https://www.tandfonline.com/loi/tbit20
Musical sonification supports visual discrimination
of color intensity
Niklas Rönnberg
To cite this article: Niklas Rönnberg (2019): Musical sonification supports visual discrimination of
color intensity, Behaviour & Information Technology, DOI: 10.1080/0144929X.2019.1657952
To link to this article: https://doi.org/10.1080/0144929X.2019.1657952
© 2019 The Author(s). Published by Informa
UK Limited, trading as Taylor & Francis
Group
Published online: 31 Aug 2019.
Submit your article to this journal
View related articles
View Crossmark data

Musical sonification supports visual discrimination of color intensity
Niklas Rönnberg
Division for Media and Information Technology, Linköping University, Linköping, Sweden
ABSTRACT
Visual representations of data introduce several possible challenges for the human visual
perception system in perceiving brightness levels. Overcoming these challenges might be
simplified by adding sound to the representation. This is called sonification. As sonification
provides additional information to the visual information, sonification could be useful in
supporting the visual perception. In the present study, usefulness (in terms of accuracy and
response time) of sonification was investigated with an interactive sonification test. In the test,
participants were asked to identify the highest brightness level in a monochrome visual
representation. The task was performed in four conditions, one with no sonification and three
with different sonification settings. The results show that sonification is useful, as measured by
higher task accuracy, and that the participant’s musicality facilitates the use of sonification with
better performance when sonification was used. The results were also supported by subjective
measurements, where participants reported an experienced benefit of sonification.
ARTICLE HISTORY
Received 13 June 2019
Accepted 10 August 2019
KEYWORDS
Interactive sonification;
musical elements;
multimodality; visual
perception; visualisation
1. Introduction
Visualisation is a common way to present research data
and share research results with other researchers, as well
as with the public. It offers a way to communicate com-
plex relations in a single glance and is convenient for
data exploration. The primary goal of visual data
exploration is to support a user in formulating questions
or hypotheses about the data. These hypotheses may be
useful for further stages of the data exploration process,
such as cluster detection, important feature detection, or
pattern and rule detection (Simoff, Bohlen, and Mazeika
2008). Seeing data visually also aids idea generation,
shows the shape of the data, possibly reveals correlations
between variables, and is a useful first step in the analysis
process (Simoff, Bohlen, and Mazeika 2008), but only if
the visual perception manages to convey the information
needed. Because, as complexity in the visual represen-
tation increases, interpretation becomes more proble-
matic and challenging. Apart from the sheer amount of
data on the visual display that might present a consider-
able difficulty for an user, there are also challenges for the
visual perception that can impair comprehension of the
visual representation.
In order to facilitate visual analysis of large data sets
and to reduce visual clutter in the representation, it is
common to use transparency renderings based on data
density (see for example Artero, deOliveira, and Levko-
witz 2004; Ellis and Dix 2007). This is typically achieved
by rendering semi-transparent objects and additively
blending them together (see an example of a parallel
coordinates plot with transparency rendering of density
in Figure 1). This method can reveal structures and
relationships in data that otherwise would have been
missed. However, using transparency renderings for
density information might be challenging for the percep-
tion, for example when perceiving simultaneous bright-
ness contrast (Ware 2013) and when distinguishing
between brightness levels. Thus, these renderings make
it difficult to observe actual numbers of blended objects
for different areas in the density representation, as well
as making it hard to find areas of similar density or
find areas of highest density.
The challenges, such as distinguishing between bright-
ness levels, related to the inherent functions of visual per-
ception can never be overcome by visualisation alone.
However, they could be addressed by adding sound as
a complementary modality to the visual representation.
The combination of the visual and the aural modalities
should make it possible to design more effective multi-
modal visual representations, as compared to when
using visual stimuli alone (Rosli and Cabrera 2015).
Sonification, the transformation of data into sound,
can be used to supplement the visual modality when a
user studies a visualisation of data to further support
understanding of the visual representation (Kramer
et al. 2010; Hermann, Hunt, and Neuhoff2011; Pinch
© 2019 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group
This is an Open Accessarticle distributed underthe terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/),
which permits non-commercial re-use, distribution, and reproduction in any medium, provided the originalwork is properly cited, and is not altered, transformed, orbuilt upon in any way.
CONTACT Niklas Rönnberg niklas.ronnberg@liu.se
BEHAVIOUR & INFORMATION TECHNOLOGY
https://doi.org/10.1080/0144929X.2019.1657952

and Bijsterveld 2012; Franinovic and Serafin2013). Tra-
ditionally, sonification is audification of data, where data
might be converted to a sound-wave or translated into
frequencies (Hermann, Hunt, and Neuhoff2011; Pinch
and Bijsterveld 2012). However, it could be questioned
to what extent this type of sonification is able to convey
information and meaning to a user. Going beyond plain
audification of data (Philipsen and Kjaergaard 2018),
sonification can be approached by deliberately designing
and composing musical sounds. Even though the con-
cept of sonification for data exploration is not new (see
for example Flowers, Buhman, and Turnage 2005),
there are few examples of studies that evaluate visualisa-
tion and sonification as a combination (see for example
Flowers, Buhman, and Turnage 1997; Nesbitt and Bar-
rass 2002; Kasakevich et al. 2007; Riedenklau, Hermann,
and Ritter 2010; Rau et al. 2015). These studies suggest
that there is a benefit of sonification in connection to
visualisation; however, few studies explored the appreci-
ation of the sounds in the sonification or the use of musi-
cal sounds. Musical sounds are here referred to as
deliberately designed and composed sounds, based on
a music-theoretical and aesthetic approach.
Sonification using musical sounds is interesting as the
use of musical elements give good control over the design
of the sounds and enables the deployment of potentially
useful musical components such as timbre, harmonics,
melody, rhythm, tempo, and amplitude (Seashore 1967;
Deliege and Sloboda 1997; Juslin and Laukka 2004; Levi-
tin 2006). Previous studies have shown promising results
for the concept of musical sonification (Ronnberg and
Johansson 2016; Ronnberg, Lundberg, and Lowgren
2016; Ronnberg et al. 2016; Ronnberg 2017,2019). As
musical sounds are well adapted, at least on a more gen-
eral level, to conveying meaning, information, and
emotions (see for example discussions in Tsuchiya, Free-
man, and Lerner 2006;Ronnberg and Lowgren 2016),
musical sonification should be a fruitful approach to
multimodal information visualisation. However, despite
various research (see examples in Kramer et al. 2010
and in Hermann, Hunt, and Neuhoff2011) it is not
clear which musical elements, or combinations of musi-
cal elements, are most suitable to use in sonification.
1.1. Aims and objectives
The aim of the current study is to investigate the benefit
of sonification using composed and deliberately designed
musical sounds compared to no sonification in the con-
text of information visualisation, to evaluate the useful-
ness (i.e. performance in terms of accuracy and
response time) of the sonification, and finally to explore
a possible effect of the user’s musicality on the benefitof
sonification. Musical elements used are: (1) a combi-
nation of Timbre and amplitude, (2) Pitch, and (3) Har-
mony. These sounds are used to interactively sonify
intensity levels in visual representations containing gra-
dients to mimic a visualisation of data.
2. Method
To explore the usefulness of sonification, and different
musical elements in the sonification, an interactive test
using musical sonification was devised to investigate: (1)
which of three conditions with sonification would be
most effective in combination with the visual represen-
tations, and (2) to what extent self-assessed musicality
would affect the usefulness of the sonification conditions.
2.1. Creation of the visual representations
The visual representations (see examples in Figure 2 and
in Figure 3) were designed to mimic cutouts of a complex
visualisation of data such as transparency renderings
based on data density (as illustrated in Figure 1), and chal-
lenge the visual perception. Similar representations can be
Figure 1. An example of a parallel coordinates plot where transparency rendering is based on the data density is used. (Illustration
courtesy by Jimmy Johansson.)
2N. RÖNNBERG

found in a variety of research disciplines, ranging from
static social science data, via medical or climate change
data, to temporal air traffic control data. As the perceptual
challenges arise due to shortcomings in the perception of
brightness levels (Ware 2013), it can be assumed that
similar difficulties will be present in a visualisation with
gradient bands. The visual representations were created
in Matlab (R2016a) using a sine wave grating. This was
done by mixing sinusoids in different frequencies, with
an addition of low-level random ripples. A triangle wave
was then multiplied with the combined wave form to cre-
ate a peak level for the highest intensity level. Ten different
output wave forms were created in this way by circularly
shifting the elements in the array containing the sinusoids,
by the randomness of the ripples, as well as by varying the
slope and magnitude of the triangle wave. As the par-
ameters were changed within sets of ten wave forms, the
difficulty level was balanced within a set of ten images.
A total of 90 images were then created. The wave form
was scaled to 8-bit integers, and the values of this grey
scale intensity map were linearly transformed to pixel
values in the green RGB channel and saved as 24-bit
RGB images in PNG format, ranging from no intensity
(black) to full intensity (pure green).
The green colour channel was chosen over red or blue
as the human visual perception is more sensible to con-
trasts in the green colour since green has higher perceived
brightness than red or blue of equal power (Smith and
Guild 1931;CIE1932). There are other colour models
better adapted to the human visual perception than
RGB, however, the use of the RGB colour model is motiv-
ated since the visual representations used in the present
study are monochromatic and intensity levels, rather
than hue or saturation, are mapped to the sonification.
2.2. Design of the sonification
SuperCollider (3.8.0) was used to create the interactive
sonification. SuperCollider is an environment and pro-
gramming language for real-time audio synthesis
(McCartney 1996,2002). In SuperCollider a synth
definition was created consisting of seven triangle
waves (see Figure 2), somewhat detuned around the fun-
damental frequency (−6, −4, −2, +2, +4, and +6
cents). The sonification was then built up by eleven
tones, creating a C-major chord (ranging from C2 to
C8, i.e. 65.41 Hz to 4186.01 Hz). This chord was mixed
with pink noise at a low sound level to create a rich har-
monic content (i.e. the timbre of the sound), yet with a
pleasant harmonic content (similar to the musical
sound used in Ronnberg 2019). A demonstration can
be found here: https://vimeo.com/261447212
2.3. Mapping between musical and visual elements
The mapping between musical and visual elements was
designed to provide three different conditions with
sonification (Timbre, Pitch, and Harmony, see Figure 2
Figure 2. The structure of the experimental setup. The sound consisted of triangle waves and noise mixed together. Pitch, Harmony,
and Timbre were adjusted according to the visual representation before the sound was output to the participant.
Figure 3. One example of the visual representations used in the test setup, showing the complexity of the grating. The grey scale
intensity map were linearly transformed to the green RGB colour channel.
BEHAVIOUR & INFORMATION TECHNOLOGY 3

and Table 1) as well as a condition with no sonification.
The values of the grey scale map created in Matlab were
transformed to different parameters in the interactive
sonification.
The first sonification setting changed the cutofffre-
quency of a band-pass filter and the amplitude of the
sound (hereafter referred to as Timbre). A soft or dull
timbre is experienced as more negative compared to a
brighter timbre (Juslin and Laukka 2004). A more com-
plex timbre is more captivating with a greater
(emotional) response as a result compared to a simpler
timbre, and a louder sound is more activating and enga-
ging compared to a less loud sound (Iakovides et al.
2004) and perception of loudness is also mapped to
brightness via amplitude (Pridmore 1992). In this con-
dition, the sound passed through a second order band-
pass filter. The cutofffrequency was mapped via a linear
to exponential conversion where the lowest intensity
level generated a cutofffrequency of 100 Hz while the
highest intensity levels yielded a cutofffrequency of
6000 Hz. The mapping between intensity levels in the
visual representation and the sonification was done line-
arly to exponentially and consequently the sonification
provided a higher level of information where the partici-
pant needed it the most to be able to provide an accurate
answer. The choice of linear to exponential mapping is
motivated by the fact that the human perception of
amplitude as well as frequency is nonlinear Everest and
Pohlmann (2015). After the band-pass filter, the sound
was mapped via a linear to exponential conversion,
where the amplitude level was almost completely attenu-
ated for the lowest intensity level, while there was no
attenuation for the highest intensity levels. Both these
musical elements, frequency content of the overtones
and amplitude, should provide potential sonic cues to
help solve the task in the test setup.
In the second sonification condition, the pitch of the
sonification was mapped to the intensity level (hereafter
referred to as Pitch). An ascending pitch is generally per-
ceived as more positive while a descending pitch is per-
ceived as more negative (Juslin and Laukka 2004), which
might correspond to the perception of brighter and dar-
ker areas in the visual representation (see for example
discussions in Bresin 2005; Palmer, Langlois, and Schloss
2016; Best 2017). Furthermore, higher pitched tones are
associated with lighter, brighter colours (Marks 1987;
Collier and Hubbard 2004; Ward, Huckstep, and Tsaka-
nikos 2006). The mapping between the intensity level in
the visual representation and the pitch of the sonification
was done linearly to exponentially for the same reason as
for the Timbre condition. At the darkest region in the
visual representation, the pitch of the sonification was
two octaves below the area with the highest intensity
level. Consequently, this sonification condition should
also be able to provide useful sonic cues for the test task.
The third sonification condition used dissonance of
the harmonic content of each tone in the sonification
(hereafter referred to as Harmony). A more complex har-
monic sound is more captivating for a listener compared
to a simpler harmonic sound (Iakovides et al. 2004), and
dissonant chords are experienced as more unpleasant
compared to harmonious major or minor chords (Palle-
sen et al. 2005). In this sonification condition, the tri-
angle waves creating each tone varied from seven tones
in unison (perfect pitch) in the area with the highest
intensity level to almost a halftone below and above
the fundamental tone (−96, −64, −32, 0, +32, +64,
+96 cents) in the lowest intensity area. For the Harmony
condition the mapping was done linearly. As the harmo-
nic components are further apart in frequency in relation
to the fundamental frequency (as is the case in the darker
areas in the visual representation) the interference
between frequencies creates a beating (Winckel 1967).
The beat frequency is equal to the difference in frequency
of the notes that interfere (Roberts 2016). Perception of
the two tones ranges from pleasant beating (when
there is a small frequency difference) to roughness
(when the difference grows larger) and eventually separ-
ation into two tones (when the frequency difference
increases even more) (Sethares 2005). As a consequence,
the beating decreases in tempo as the harmonic com-
ponents come closer in frequency, and at the brightest
vertical pixel column the beating stops and all harmonic
components lock to the fundamental frequency. The
physical behaviour of the frequencies involved creates a
clear sonification clue that makes even small differences
in harmonic content rather easily detectable. Conse-
quently, the harmonic complexity of the sonification
should provide sonic cues for the participants to solve
the tasks in the test.
2.4. Participants
For the present study, 25 students at Linköping Univer-
sity (14 female and 11 male) with a median age of 22
(range 18–31) with normal, or corrected to normal,
vision and self-reported normal hearing were recruited.
No compensation for participating in the study was
provided.
Table 1. The mapping between the sonification and the visual
representation for the three sonification conditions.
Sonification condition Low brightness High brightness
Timbre Attenuated, more bass Louder, more treble
Pitch Low pitched tone High pitched tone
Harmony Much dissonance Perfect harmony
4N. RÖNNBERG

2.5. Experimental design and procedure
An interactive test was devised to explore a possible
benefit of sonification (see Figure 2), and the effects of
the different sonification conditions. The test session
took 20 min at the most, and was initiated with learning
trials for familiarising and to reduce learning effects. The
learning trials consisted of all four sonification con-
ditions. After the training, the test was divided into
four parts according to the four sonification conditions
with 20 visual representations in each, and a short
break after each part where the participants answered a
questionnaire about the particular sonification con-
dition. The order of sonification conditions was balanced
between subjects to avoid order effects.
The participants moved a slider by using the compu-
ter mouse, to mark the vertical pixel column with the
highest brightness level in the visual representation,
and the sonification was adjusted according to the inten-
sity level for that pixel column (see Figure 3). The partici-
pants were asked to answer as quickly as possible. After
marking a vertical pixel column, the participants pressed
the large button beneath the slider and the next trial was
automatically initiated. The accuracy for each trial was
measured as the absolute difference between the highest
intensity level in the visual representation and the par-
ticipant’s marked brightness level. Hence, a lower
measure was equal to higher accuracy. The response
time was also recorded. For the statistical analyses, the
overall accuracy was calculated as the mean error for
the 20 answers in each sonification condition, and the
response time was the mean response time for each
sonification condition. Accordingly, the experiment
yielded both objective measures of sonification, accuracy
and response time, and subjective measures from a
questionnaire.
SuperCollider was used on a MacBook Pro, presenting
visual stimuli on a 21′′ computer screen and auditory
stimuli via a Universal Audio Apollo Twin sound inter-
face through a pair of Beyerdynamic DT-770 Pro head-
phones. The headphones provided an auditory
stimulation of approximately 65 dB SPL. A quiet office
was used for the test, and even if there were some ambi-
ent sounds, the test environment was deemed quiet
enough not to affect the tests conducted.
2.6. Questionnaire
A questionnaire was used to record subjective data to
complement the objective measures. In the beginning
of the test the participants were asked to rate their musi-
cality via a 5-point Likert scale from 1 (Not very exten-
sive) to 5 (Very extensive). After each sonification
condition (No sonification, Timbre, Pitch, Harmony)
the participant answered questions about the difficulty
level they experienced in finding the brightest vertical
pixel column, and if they experienced a benefit of sonifi-
cation (if sonification was used) in terms of accuracy and
response time. Answers in the questionnaire were given
ranging from 1 (Strongly disagree) to 5 (Strongly agree).
Finally, after a total of 90 trials, the participants answered
questions regarding if they experienced an overall benefit
of the sonification or not.
3. Results
The participants were divided into two groups: Low
musicality (n=12) for ratings from 1 to 3, and High musi-
cality (n=13) for ratings 4 and 5. According to Kolmo-
gorov–Smirnov tests the data were not normally
distributed, thus non-parametric tests were used to ana-
lyse the data. Bonferroni correction for multiple com-
parisons was applied as appropriate. Descriptive
statistics can be found in Table 2.
Accuracy was measured in terms of the mean errors
made. A Friedman test showed a significant difference
in mean errors between the four conditions
(
x
2(3) =38.57, p,0.001). Dunn-Bonferroni post-
hoc tests showed only significant differences between
No sonification and the three conditions with sonifi-
cation; Timbre (p=0.001), Pitch (p<0.001), and Har-
mony (p<0.001), where there were less errors when
sonification was used. However, no statistically signifi-
cant differences were found between the three conditions
with sonification. Mann–Whitney Utests showed signifi-
cantly less errors for the High musicality group for Tim-
bre (U=41.5, p=0.046), for Pitch (U=23.5, p=0.002 ), and
for Harmony (U=20.0, p=0.001 ), but there were no sig-
nificant difference between the groups for the No sonifi-
cation condition.
For response time, a Friedman test showed significant
differences between the four conditions
(
x
2(3) =42.81, p,0.001). Dunn-Bonferroni post-
hoc tests showed only significant differences between
No sonification and the three conditions with sonifi-
cation; Timbre (p<0.001), Pitch (p<0.001), and Har-
mony (p<0.001), where response time were longer
Table 2. Descriptive statistics with mean errors and response
time measurements (in seconds) for Low musicality and High
musicality. Standard deviation in parentheses.
No sonification Timbre Pitch Harmony
Low musicality 13.9 (5.4) 4.0 (4.1) 3.5 (2.0) 3.6 (2.2)
High musicality 11.4 (9.4) 2.2 (1.2) 1.3 (0.8) 0.7 (0.8)
Response time No sonification Timbre Pitch Harmony
Low musicality 4.7 (1.6) 10.4 (4.0) 11.9 (4.5) 11.7 (4.6)
High musicality 8.6 (3.2) 14.9 (7.0) 16.9 (9.1) 15.9 (7.3)
BEHAVIOUR & INFORMATION TECHNOLOGY 5

when sonification was used. There were no statistically
significant differences between the three sonification set-
tings. Mann–Whitney Utests showed significantly
longer response time for the High musicality group in
the condition with No sonification (U=24.0, p=0.002),
but there were no significant difference between the
groups for the conditions with sonification.
The subjective measures from the questionnaire
showed that the participants generally experienced
sonification as helpful, see Figure 6. The median ranking
(1 =`Veryhard′to 5 =`Veryeasy′) for difficulty in No
sonification was 2 (range: 1–4), and in Sonification it
was 4 (range: 3–5). These results suggest that the task
was experienced as easier with sonification than without.
The experienced difficulty for No sonification as well as
for Sonification was similar for both groups (Low musi-
cality and High musicality). The experienced help from
sonification for improving accuracy was also measured
(1 =`Nohelpatall′to 5 =`Muchhelp′). The median rat-
ing was 5 (range: 4–5), which suggests that
the participants experienced a benefit of sonification.
The experienced benefit of sonification was high and
similar for both groups (Low musicality and High
musicality).
Finally, the experienced benefit of sonification for giv-
ing a faster response was measured (1 =`Muchslower′
to 5 =`Muchfaster′), where the median rating was 4
(range: 1–5), suggesting that most participants experi-
enced that sonification supported them in giving faster
responses. There were some differences between the
groups, which suggest that participants in the Low musi-
cality group generally perceived the sonification to sup-
port in giving a faster responses, while the High
musicality group had a more diverse impression.
4. Discussion
4.1. Accuracy
The results found in the present study suggest that sonifi-
cation can improve perception of colour brightness (see
Table 2 and Figure 4). The additional information intro-
duced by the sonification made it possible for the partici-
pants to improve their accuracy when the information in
the visual modality was insufficient for giving an answer
with high accuracy. Consequently, the sonification sup-
ported the visual perception in the task. This interpret-
ation of the results was also supported by subjective
measurements. Results from the questionnaire suggested
that the difficulty level of the task was reduced and that
the participants experienced the sonification as very
helpful in improving the accuracy of their answers.
4.2. Response time
Response time was found to be longer when sonification
was used, compared to the No sonification condition (see
Table 2 and Figure 5). This indicates that the participants
used the extra information provided by the sonification
to refine their selection in the test, reaching a higher
accuracy, and that this procedure took longer time.
Interestingly, when considering the subjective ratings
many of the participants stated that they experienced
sonification to improve their response time as well as
the accuracy, which was not the case according to the
measured response times. It might be hypothesised
that some participants believed that they performed the
task faster, as they might have experienced the compari-
son between areas in the visual representation easier
when sonification was used. Furthermore, as the amount
Figure 4. Generally, the mean error decreased when sonification was used. The High musicality group made less errors compared to the
Low musicality group.
6N. RÖNNBERG

of information was increased when sonification was
used, the task became more demanding from a percep-
tual perspective, which in turn made the participants
more deeply engaged in the task. In general, when some-
one is more engaged in a task, time is perceived to pass
more quickly (Conti 2001; Chastona and Kingstone
2004; Sackett et al. 2010). This well-known phenomenon
may be an explanation for the discrepancy between sub-
jective experience and objective measures with regard to
response times. The results show that sonification is use-
ful in terms of higher accuracy, but this comes at the
price of longer response times. It could be argued that
in situations where accuracy is more important than
response time, then sonification as used in the present
study is useful. For example when a researcher is explor-
ing an interactive multimodal visualisation for finding
relationships in the data to gain new insights in a
research question, or when sonification is used for clar-
ifying a user interaction in an educational situation.
4.3. The participants’musicality
Even if the groups were small, musicality had effects
on the results. The High musicality group had statisti-
cally significant higher accuracy in the conditions with
sonification compared to the Low musicality group
(see Table 2,Figures 4, and 5). The results suggest
that the High musicality group used their experience
and knowledge of musical sounds to reach a higher
accuracy. Interestingly, the High musicality group
had statistically significant longer response times in
the condition with No sonification, further work
needs to be done to explore if this result is repeatable
and what the causes might be.
4.4. Sonification condition
There were no statistical significant differences between
the conditions with sonification (Timbre, Pitch, or Har-
mony). This indicates that regardless of the specific
musical element used in the sonification, and regardless
of the participant’s musicality, accuracy increased when
sonification was used. These results are promising, as
proficiency in music theory should not be required to
hear the differences in a sonification. However, when
studying mean and confidence intervals for accuracy
there might be a trend discernible for the High musical-
ity group, where Harmony had higher accuracy com-
pared to Pitch, which in turn had higher accuracy
compared to Timbre (see Table 2 and Figure 4). A simi-
lar trend was not present for the Low musicality group,
where accuracy was more or less equal for all conditions
with sonification. This might suggest that for partici-
pants with high musicality, who knew what musical
cues to listen for, differences in Harmony and in Pitch
provided stronger cues than Timbre. If this is the case,
the use of such musical elements benefits participants
with higher musicality more, compared to participants
with lower musicality.
4.5. The visual representation and
experimental task
The visual representations used in the experimental
setup within the present study contained visual elements,
i.e. differences in intensity levels, that challenged the per-
ception of brightness levels. The visual representations
used could therefore be seen as selections from a larger,
more complex and real visual representation used for
data exploration, where misconceptions due to
Figure 5. The mean response time was longer when sonification was used, and the High musicality group had longer response times
compared to the Low musicality group.
BEHAVIOUR & INFORMATION TECHNOLOGY 7

shortcomings in the visual perception could be a real and
relevant drawback in interpretation of the data. Conse-
quently, results found in, and the knowledge gained
within, the present study should be generalisable to
other visual representations as well.
Finding the highest brightness level in a data set might
be solved using a mathematical operation. However, this
is true if the user already knows what he or she is looking
for in the data. The task used in the present study can
therefore be considered a good simplification that
enabled the examination of musical sonification and
visual challenges in a controlled setting.
4.6. A possible learning effect
The highest intensity level was the same in all visual rep-
resentations used in the test setup (i.e. 255 in the green
RGB channel). Finding the brightest vertical pixel col-
umn could consequently have been facilitated by mem-
orising the Timbre, Pitch, or Harmony from the
previous trial and comparing it with the sonification in
the current trial. The echoic memory is the sensory
memory for sounds that have just been perceived (Carl-
son et al. 2009), and it is capable of storing auditory
information for a short period of time. The stored
sound resonates in the mind and is replayed for 3-4 s
after the presentation of auditory stimuli (Radvansky
2005). The echoic memory encrypts moderately primi-
tive aspects of the sound, such as pitch (Strous, Cowan,
and Ritter 1995). Thus, the echoic memory could help
in finding the brightest vertical pixel column, as this pos-
ition would sound ”right”in the participant’s mind. This
reasoning suggests that the learning effect, if present,
might thus have made the sonification to provide
additional information, as well as making a comparison
with the sound kept in memory possible. This can be
seen as something useful, as this suggests that with learn-
ing how to use sonification, performance can be
increased.
5. Conclusion
The present study evaluated the usefulness of sonifi-
cation as a complement to visual representations. The
results show that there was a benefit of sonification, in
terms of increased accuracy, in selecting the vertical
pixel column with the highest colour brightness in the
visual representations. This suggests that sonification
facilitated perception of colour brightness, and helped
users overcome challenges for the visual perception in
the visual representations. This result was also supported
by the subjective measurements where an experienced
benefit of sonification was reported. However, the use
and processing of the additional information provided
by the sonification took time, leading to a longer
response time when sonification was used compared to
the No sonification condition. This suggests that there
is a speed/accuracy trade-offwhere the usefulness
might decrease in situations where fast response times
is of the essence. Finally, there was an effect of musicality
in the statistical analysis, where participants with higher
musicality had higher accuracy in the test conditions
with sonification.
6. Future work
For future work, further musical elements such as tempo
and rhythm would be interesting to explore. Also, the
Figure 6. The subjective measures indicate that sonification made the task more easy and was experienced as helpful, but overall not as
helpful for giving faster responses.
8N. RÖNNBERG

combination of musical elements such as amplitude and
pitch, or harmony and timbre, could be deployed to
investigate whether the combination could provide
even stronger sonic cues, and if it is possible to provide
different sonic cues simultaneously by using different
musical elements. The application of musical sonifi-
cation, using a musical theoretical approach, could also
be evaluated in relation to the more classical form of
purely data-driven sonification. These questions should
be evaluated in relation to standardised tests of the par-
ticipant’s musicality and music perception skills (Law
and Zentner 2001), to further explore to what extent
an individual’s musicality affects the perception of
sonification.
Furthermore, it would be intriguing to evaluate sonifi-
cation support for a wider range of visual representations
and the use of real data, particularly with domain
experts, and in relation to (for example) the Visual Infor-
mation Seeking Mantra (Shneiderman 1996). Sonifi-
cation could be studied as a way of creating an
overview of an entire collection of data, or as a way to
support the examination of relationships among data
items. The information visualisation mantra provides a
scaffold for further studies of the usefulness of sonifi-
cation in visual data exploration and information seek-
ing. Data for further studies could, for example, be
obtained from bio-sensors used in the medical sciences,
time cycles and activities in the social sciences, or climate
change data. The use of real data and different visualisa-
tion techniques would indicate which musical elements
in the sonification that are most suitable to use interac-
tively in combination with which type of visualisation
technique. These future inquires would generate an
understanding of the implications of the sonification
research, as well as suggest areas where sonification
would be useful as an additional tool in visual data
exploration.
Disclosure statement
No potential conflict of interest was reported by the author.
ORCID
Niklas Rönnberg http://orcid.org/0000-0002-1334-0624
References
Artero, A. O., M. C. F. de Oliveira, and H. Levkowitz. 2004.
“Uncovering Clusters in Crowded Parallel Coordinates
Visualizations.”In Proc. IEEE Symposium on Information
Visualization INFOVIS ’04,81–88. Washington, DC:IEEE
Computer Society. doi:10.1109/INFOVIS.2004.68.
Best, J. 2017.Colour Design: Theories and Applications. 2nd ed.
Duxford: Elsevier Ltd., Woodhead Publishing.
Bresin, R. 2005.“What is the Color of that Music
Performance?”In Proc. International Computer Music
Conference (ICMC) 2005, 367–370. San Francisco, CA:
International Computer Music Association.
Carlson, N. R., D. Heth, H. Miller, J. Donahoe, and G. N.
Martin. 2009.Psychology: The Science of Behavior. Harlow:
Pearson.
Chastona, A., and A. Kingstone. 2004.“Time Estimation: The
Effect of Cortically Mediated Attention.”Brain and
Cognition 55: 286–289.
CIE. 1932.Commission Internationale de l’Eclairage
Proceedings, 1931. Cambridge: Cambridge University Press.
Collier, W. G., and T. L. Hubbard. 2004.“Musical Scales and
Brightness Evaluations: Effects of Pitch, Direction, and
Scale Mode.”Musicae Scientiae 8: 151–173.
Conti, R. 2001.“Time Flies: Investigating the Connection
Between Intrinsic Motivation and the Experience of
Time.”Journal of Personality 69: 1–26.
Deliége, I., and J. Sloboda. 1997.Perception and Cognition of
Music. Hove: Psychology Press Ltd.
Ellis, G., and A. Dix. 2007.“ATaxonomy of Clutter Reduction
for Information Visualisation.”IEEE Transactions on
Visualization and Computer Graphics 13: 1216–1223.
Everest, F. A., and K. C. Pohlmann. 2015.Master Handbook of
Acoustics. 6th ed. New York, NY: McGraw-Hill Education
LLC.
Flowers, J. H., D. C. Buhman, and K. D. Turnage. 1997.“Cross-
Modal Equivalence of Visual and Auditory Scatterplots for
Exploring Bivariate Data Samples.”Human Factors 39:
341–351.
Flowers, J. H., D. C. Buhman, and K. D. Turnage. 2005.“Data
Sonification From the Desktop: Should Sound Be Part of
Standard Data Analysis Software?.”ACM Transactions on
Applied Perception 2: 467–472.
Franinovic, K., and S. Serafin. 2013.Sonic Interaction Design.
Cambridge, MA: MIT Press.
Hermann, T., A. Hunt, and J. G. Neuhoff.2011.The
Sonification Handbook. 1st ed. Berlin: Logos Publishing
House.
Iakovides, S. A., V. T. H. Iliadou, V. T. H. Bizeli, S. G. Kaprinis,
K. N. Fountoulakis, and G. S. Kaprinis. 2004.
“Psychophysiology and Psychoacoustics of Music:
Perception of Complex Sound in Normal Subjects and
Psychiatric Patients.”Annals of General Hospital
Psychiatry 3: 1–4.
Juslin, P. N., and P. Laukka. 2004.“Expression, Perception,
and Induction of Musical Emotions: A Review and a
Questionnaire Study of Everyday Listening.”Journal of
New Music Research 33: 217–238.
Kasakevich, M., P. Boulanger, W. F. Bischof, and M. Garcia.
2007.“Augmentation of Visualisation Using Sonification:
A Case Study in Computational Fluid Dynamics.”In Proc.
IPT-EGVE Symposium,89–94. Germany, Europe: The
Eurographics Association.
Kramer, G., B. Walker, T. Bonebright, P. Cook, J. H. Flowers,
N. Miner, and J. Neuhoff.2010.“Sonification Report: Status
of the Field and Research Agenda. Vol. 444, 1–29. Faculty
Publications, Department of Psychology.
Law, L. N. C., and M. Zentner. 2001.“Assessing
Musical Abilities Objectively: Construction and Validation
BEHAVIOUR & INFORMATION TECHNOLOGY 9

of the Profile of Music Perception Skills.”PLoS ONE 7:
1–15.
Levitin, D. J. 2006.This is Your Brain on Music: The
Science of a Human Obsession.NewYork:Dutton/
Penguin Books.
Marks, L. E. 1987.“On Cross-modal Similarity: Auditory-
visual Interactions in Speeded Discrimination.”Journal of
Experimental Psychology: Human Perception and
Performance 13: 384–394.
McCartney, J. 1996.“SuperCollider: A New Real-Time
Synthesis Language.”In Proc. International Computer
Music Conference (ICMC), 257–258. Hong Kong:
Michigan Publishing.
McCartney, J. 2002.“Rethinking the Computer Music
Language: SuperCollider.”IEEE Computer Graphics &
Applications 26: 61–68.
Nesbitt, K. V., and S. Barrass. 2002.“Evaluation of a
Multimodal Sonification and Visualisation of Depth of
Market Stock Data.”In Proc. International Conference on
Auditory Display (ICAD),2–5. International Community
on Auditory Display.
Pallesen, K. J., E. Brattico, C. Bailey, A. Korvenoja, J. Koivisto,
A. Gjedde, and S. Carlson. 2005.“Emotion Processing of
Major, Minor,and Dissonant Chords: A Functional
Magnetic Resonance Imaging Study.”Annals New York
Academy of Sciences 1060: 450–453.
Palmer, S. E., T. A. Langlois, and K. B. Schloss. 2016.
“Music-to-Color Associations of Single-Line Piano
Melodies in Non-synesthetes.”Multisensory Research 29:
157–193.
Philipsen, L., and R. S. Kjærgaard. 2018.The Aesthetics of
ScientificData Representation: More Than Pretty Pictures:
Routledge Advances in Art and Visual Studies. New York:
Routledge.
Pinch, T., and K. Bijsterveld. 2012.The Oxford Handbook of
Sound Studies. Oxford: Oxford University Press.
Pridmore, R. W. 1992.“Music and Color: Relations in the
Psychophysical Perspective.”Color Research & Application
17: 57–61.
Radvansky, G. 2005.Human Memory. Boston: Allyn and
Bacon.
Rau, B., F. Frieß, M. Krone, C. Müller, and T. Ertl. 2015.
“Enhancing Visualization of Molecular Simulations using
Sonification.”In Proc. IEEE 1st International Workshop on
Virtual and Augmented Reality for Molecular Science
(VARMS@IEEEVR 2015),25–30. Arles: The Eurographics
Association.
Riedenklau, E., T. Hermann, and H. Ritter. 2010.“Tangible
Active Objects and Interactive Sonification as a Scatter
Plot Alternative for the Visually Impaired.”In Proc. 16th
International Conference on Auditory Display (ICAD
2010),1–7. Germany, Europe; International Community
for Auditory Display.
Roberts, G. E. 2016.From Music to Mathematics: Exploring the
Connections. Baltimore: Johns Hopkins University Press.
Rönnberg, N. 2017.“Sonification Enhances Perception of
Color Intensity.”In Proc. IEEE VIS Infovis Posters
(VIS2017),1–2. Phoenix, AZ: IEEE VIS.
Rönnberg, N. 2019.“Sonification Supports Perception of
Brightness Contrast.”Journal on Multimodal User
Interfaces 1–9. doi:10.1007/s12193-019-00311-0
Rönnberg, N., G. Hallström, T. Erlandsson, and J. Johansson.
2016.“Sonification Support for Information Visualization
Dense Data Displays.”In Proc. IEEE VIS Infovis Posters
(VIS2016),1–2. Baltimore, MD: IEEE VIS.
Rönnberg, N., and J. Johansson. 2016.“Interactive Sonification
for Visual Dense Data Displays.”In Proc. 5th Interactive
Sonification Workshop (ISON-2016),63–67. Germany:
CITEC, Bielefeld University.
Rönnberg, N., and J. Löwgren. 2016.“The Sound Challenge to
Visualization Design Research.”In Proc. EmoVis 2016,
ACM IUI 2016 Workshop on Emotion and Visualization,
Linköping Electronic Conference Proceedings, Vol. 103,
31–34. Sweden.
Rönnberg, N., J. Lundberg, and J. Löwgren. 2016.“Sonifying
the Periphery: Supporting the Formation of Gestalt in Air
Traffic Control.”In Proc. 5th Interactive Sonification
Workshop (ISON-2016),23–27. Germany: CITEC,
Bielefeld University.
Rosli, M. H. W., and A. Cabrera. 2015.“Gestalt Principles in
Multimodal Data Representation.”IEEE Computer
Graphics & Applications 35: 80–87.
Sackett, A. M., T. Meyvis, L. D. Nelson, B. A. Converse, and A.
L. Sackett. 2010.“You’re Having Fun When Time Flies: The
Hedonic Consequences of Subjective Time Progression.”
Psychological Science 21: 111–117.
Seashore, C. E. 1967.Psychology of Music. New York: Dover.
Sethares, W. A. 2005.Tuning, Timbre, Spectrum, Scale. 2nd ed.
London: Springer.
Shneiderman, B. 1996.“The Eyes Have It: A Task by Data
Type Taxonomy for Information Visualizations.”In Proc.
IEEE Symposium on Visual Languages, 336–343.
Washington: IEEE Computer Society Press.
Simoff, S., M. Bohlen, and A. Mazeika. 2008.Visual Data
Mining: Theory, Techniques and Tools for Visual
Analytics. Berlin, New York: Springer.
Smith, T., and J. Guild. 1931.“The C.I.E. Colorimetric
Standards and Their Use.”Transactions of the Ical Society
33: 73–134.
Strous, R. D., N. Cowan, W. Ritter, and D.C. Javitt, 1995.
“Auditory Sensory (ëchoic) Memory Dysfunction in
Schizophrenia.”The American Journal of Psychiatry 152:
1517–1519.
Tsuchiya, T., J. Freeman, and L. W. Lerner. 2006.“Data-To-
Music API: Real-Time Data-Agnostic Sonification with
Musical Structure Models.”In Proc. 21st International
Conference on Auditory Display (ICAD 2015), 244–251.
Graz, Styria: Georgia Institute of Technology.
Ward, J., B. Huckstep, and E. Tsakanikos. 2006.“Sound-colour
Synaesthesia: To what Extent Does it Use Cross-modal
Mechanisms Common to Us All?.”Cortex 42: 264–280.
Ware, C. 2013.Information Visualization: Perception for
Design. 3rd ed. San Francisco: Morgan Kaufmann
Publishers Inc.
Winckel, F. 1967.Music, Sound and Sensation: A Modern
Exposition. New York: Dover Publications, Inc.
10 N. RÖNNBERG