Conference PaperPDF Available

Two-dimensional psychoacoustic sonification (Precis)

Authors:

Abstract and Figures

A psychoacoustically-motivated sonification principle is introduced and experimentally verified. It is able to translate two orthogonal input dimensions by means of auditory qualities, which are orthogonal in perception as well. After around five minutes of explanations and passive training, six out of seven subjects were able to interpret the presented data fairly well in a passive listening test. For the future, an experiment with interactive training and interactive tasks is planned.
Content may be subject to copyright.
!"#$%&'()*&#)+,- .*/01#+0#2*3&0- *#$
)&4&0+3&#)-
Tim$Ziemer,$University$of$Bremen,$Bremen$Spatial$Cognition$
Center,$Bremen,$Germany$
Abstract$
A$ psychoacoustically-motivated$ sonifi-
cation$principle$is$introduced$and$exper-
imentally$ verified.$ It$ is$ able$ to$ translate$
two$ orthogonal$ input$ dimensions$ by$
means$ of$ auditory$ qualities,$ which$ are$
orthogonal$ in$ perception$ as$ well.$ After$
around$five$minutes$of$explanations$and$
passive$ training,$ six$ out$ of$ seven$ sub-
jects$were$able$to$interpret$the$ present-
ed$data$ fairly$well$in$ a$passive$ listening$
test.$For$ the$ future,$an$ experiment$ with$
interactive$training$and$interactive$tasks$
is$planned.$
$
1.$Background$
Most$ sonification$ approaches$ map$ or-
thogonal$input$data$directly$to$individu-
al$ audio$ parameters,$ like$ horizontal$ de-
flection$ to$ fundamental$ frequency$ and$
vertical$ deflection$ to$ amplitude.$ Unfor-
tunately,$ this$ strategy$ neglects$ the$ fact$
that$ each$ audio$ parameter$ may$ affect$
several$ perceptual$ qualities.$ Loudness,$
for$example,$is$largely$affected$by$ampli-
tude,$but$also$by$frequency$and$spectral$
distribution.$ Likewise,$ pitch$ is$ affected$
mostly$ by$ fundamental$ frequency$ and$
periodicity,$ but$ also$ by$ level$ and$ spec-
tral$distribution$(Fastl$&$Zwicker,$2007).$
As$ a$ consequence,$ orthogonal$ input$ di-
mensions$are$ not$ orthogonal$in$ percep-
tion$and$the$sonification$is$prone$to$mis-
interpretation$and$confusion.$
$
2.$Aim$and$Question$
In$ this$ study,$ a$ psychoacoustic$ sonifica-
tion$core$is$presented$and$experimental-
ly$ verified.$ It$ allows$ mapping$ two$ or-
thogonal$input$ dimensions$ to$individual$
aspects$of$sound$perception.$It$is$easy$to$
learn$ and$ readily$ interpretable.$In$ the$
tested$ scenario,$ the$ sound$ represents$
the$ location$ of$ a$ target$ in$ two-
dimensional$space.-
-
3.$Method$
A$ qualitative$ depiction$ of$ the$ sonifica-
tion$ principle$ is$ illustrated$ in$ Fig.$ 1.$ It$
helps$understanding$ the$verbal$ descrip-
tion:$ Horizontal$ direction$ is$ mapped$ to$
direction$of$chroma$change$of$a$Shepard$
tone$(Shepard,$1964).$If$the$target$lies$to$
the$ left,$ the$ chroma$ changes$ counter-
clockwise,$ if$ the$ target$ lies$ to$ right,$ it$
changes$ clockwise.$ This$ creates$ the$ au-
ditory$ illusion$ of$ an$ infinitely$ falling$ or$
rising$ pitch.$ The$ horizontal$ distance$ is$
mapped$ to$ the$ speed$ of$ the$ chroma$
change.$ Perceptually,$ this$ means:$ the$
further$ away,$ the$ faster$ the$ perceived,$
infinite$ glissando.$ The$ vertical$ dimen-
sion$is$divided$into$two$half-dimensions.$
If$ the$ target$ lies$ above,$ the$ distance$ is$
mapped$ to$ the$ degree$ of$ inharmonicity,$
roughness,$and$noisiness.$If$it$lies$below,$
distance$is$mapped$to$speed$of$loudness$
fluctuations,$ i.e.,$ beating.$ Details$ on$ the$
technical$implementation$of$this$percep-
tual$ mapping$ can$ be$ found,$ e.g.,$ in$
(Ziemer$&$Black,$2017a).$
$
5&627(- 89- :2+,&3+3&;(- %(.&03&#)- #4- 31(- *#)&4&0 +3&#)-
.7&)0&.,(<-
A$simple$listening$test$was$conducted$to$
evaluate$the$psychoacoustic$sonification$
approach.$ After$ a$ short$ explanation$ of$
the$ sonification$ principle,$ including$
o<<<<<<>>> > > > pitch
noisiness
beating
some$ audio$ examples,$ seven$ subjects$
identified$one$out$ of$ 16$fields$ on$ a$two-
dimensional$map$in$19$trials.$The$map$is$
illustrated$in$Fig.$ 2.$ The$target$field$was$
indicated$my$means$of$the$psychoacous-
tic$ sonification.$ Target$ fields$ lay$ either$
slightly$ or$ far$ to$ the$ left/right,$ and$
slightly$ or$ far$ above/below$ the$ origin.$
The$test$is$not$interactive.$Sounds$repre-
senting$a$static$target$were$played$to$the$
subjects.$ None$ of$them$ had$ previous$
experience$with$sonification.$
$
5&627(-=9- >+.- "&31-8?-4&(,%*<-!1(-07#**1+&7-*1#"*-
31(-#7&6&)-+)%-31(-+@(*<-!1(-4&(,%*-+7(A-(<6<A-*,&613,/-
+B#;(- +)%- *,&613,/- 3#- 31(- 7&613- C8DA- *,&613,/- +B#;(-
+)%- 4+7- 3#- 31(- 7&613- C=DA- 4+7- +B#;(- +)%- *,&613,/- 3#-
31(-7&613-CED-+)%-4+7-+B#;(-+)%-4+7-3#-31(-7&613-CFD<-
4.$Results$
In$ 83%$ of$ all$ trials,$ the$ cardinal$ direc-
tion$ of$ the$ target$ field$ was$ identified$
correctly,$ i.e.,$ north-east,$ south-east,$
south-west$ and$ north-west.$ The$ exact$
target$ field$ was$ identified$ correctly$ in$
41%$ of$ all$ trials.$ This$ is$ significantly$
higher$than$the$1/19$ =$ 6%$ chance$ level.$
The$correct$left/right$direction$was$cho-
sen$ in$ 91%$ of$ all$ trials,$ the$ correct$
up/down$direction$in$89%.$One$musical-
ly$trained$subject$identified$the$cardinal$
direction$ correctly$ in$ 95%,$ the$ exact$
target$ field$ 63%$ of$ all$ trials,$ and$ made$
one$ left/right$ and$ no$ up/down$ error.$
Another$ subject,$ however,$ performed$
only$slightly$better$than$chance.$
5.$Conclusion$
Inexperienced$ users$ interpret$ the$ two-
dimensional$psychoacoustic$ sonification$
fairly$ accurately.$ Already$ after$ some$
minutes$ of$ introduction,$ subjects$ were$
able$ to$ interpret$ the$ sonified$ infor-
mation$precisely$ in$ up$to$ 63%$ of$all$ tri-
als$ and$ to$ keep$ orthogonal$ dimensions$
apart$ in$ up$ to$ 95%$ of$ all$ trials.$ In$ the$
future,$ the$ approach$ will$ be$ verified$ in$
an$ interactive$ task,$ and$ psychoacoustoc$
models$will$serve$for$a$ metric$ scaling$of$
the$perceptual$dimension.$An$interactive$
demonstration$of$the$current$state$of$the$
psychoacoustic$sonification$will$be$given$
at$the$conference,$showing$its$benefit$for$
gesture$ analysis$ and$ blind$ guidance$
tasks.$One$ interactive$demo$ of$the$ soni-
fication$ has$ already$ been$ given$ at$ the$
Acoustics( '17$ meeting$ in$ Boston,$ MA$
(Ziemer$&$Black,$ 2017b).$By$the$ time$ of$
the$ conference,$ a$ thorough$ description$
of$ the$ sonification$ principle,$ the$ listen-
ing$ test$ setup$ and$ the$ results$ will$ be$
available$in$(Ziemer$et$al.,$2017).$
$
References$
(Fastl$&$ Zwicker,$ 2007)$ H.$ Fastl$ &$ E.$
Zwicker:$ Psychoacoustics.( Facts( and(
Models,$3rd$ed.,$Springer$2007.$
(Shepard,$ 1964)$ R.$ N.$ Shepard,$ "Circu-
larity$in$Judgments$of$Relative$Pitch",$in:$
J.( Acoust.( Soc.( Am.( 36(12),$ 1964,$ pp.$
23462353.$
(Ziemer$et$al.,$ 2017)$T.$Ziemer,$ D.$Black$
&$ H.$ Schultheis,$ "Psychoacoustic$ Sonifi-
cation$ Design$ for$ Navigation$ in$ Surgical$
Interventions",$in:$Proc.(Mtgs.(Acoust.$30,$
2017.$
(Ziemer$&$ Black,$2017b)$ T.$ Ziemer$ &$ D.$
Black,$ "Psychoacoustic$ sonification$ for$
tracked$medical$instrument$guidance",$J.$
Acoust.$Soc.$Am.$141(5),$2017,$p.$3694.$
(Ziemer$&$ Black,$ 2017a)$ T.$ Ziemer$ &$ D.$
Black,$ "Psychoacoustically$ motivated$
sonification$ for$ surgeons",$ IJCARS$ 12$
(Suppl$1):1,$2017,$pp.$265266.$

Supplementary resource (1)

... The functionality of these two dimensions has already been demonstrated in experiments with passive listeners [24,25] and active users [3,4]. Already passive users were able to attribute a sonified target to the correct region on a map with 16 regions. ...
... At the moment we are carrying out experiments with passive listeners. The experiment resembles the one that we have carried out to evaluate our two dimensional sonification [24,25]. The results of the previous experiments demonstrated that these two dimensions are in fact readily interpretable and orthogonal. ...
... Group 1 is a reference group. They solve the task in the x-y-plane, illustrated in Fig. 2. The experiment resembles the experiment described in [24,25]. Group 2 solves the task in x-z-plane, i.e., they tests the new dimension against the xdimension as shown in Fig. 3. Group 3 solves the task in z-y-plane, i.e., they tests the new dimension against the ydimension as as illustrated in Fig. 4. sented and discussed in the following sections. ...
Conference Paper
Full-text available
Sonification is the communication of variable data by means of sound. It can be used as an alternative or augmentation of visualization. In this paper we describe and evaluate a sonification approach that communicates the location of a target in three dimensional space. This is achieved by thorough consideration of auditory perception in our soni-fication. An experiment with passive listeners reveals that the sonification is readily interpretable. Listeners are able to identify the direction and distance along the single dimensions. This renders the sonification suitable for interactive navigation tasks in three dimensional space.
... Psychoacoustic sonification [27,29,30] is the first sonification technology that satisfies all of the above-listed demands. Psychoacoustics describes the relationships between the physical, acoustic quantities and the perceptual, auditory qualities of sound by means of nonlinear equations [28,39]. ...
Article
Full-text available
When operating a machine, the operator needs to know some spatial relations, like the relative location of the target or the nearest obstacle. Often, sensors are used to derive this spatial information, and visual displays are deployed as interfaces to communicate this information to the operator. In this paper, we present psychoacoustic sonification as an alternative interface for human-machine interaction. Instead of visualizations, an interactive sound guides the operator to the desired target location, or helps her avoid obstacles in space. By considering psychoacoustics-i.e., the relationship between the physical and the perceptual attributes of sound-in the audio signal processing, we can communicate precisely and unambiguously interpretable direction and distance cues along three orthogonal axes to a user. We present exemplary use cases from various application areas where users can benefit from psychoacoustic sonification.
... This means, when moving the vector X, it is impossible to recognize whether the xvalue, the y-value, or both has/have changed. Psychoacoustic sonification [27,29,30] is the first sonification technology that satisfies all of the above-listed demands. Psychoacoustics describes the relationships between the physical, acoustic quantities and the perceptual, auditory qualities of sound by means of nonlinear equations [28,41]. ...
Preprint
Full-text available
When operating a machine, the operator needs to know some spatial relations, like the relative location of the target or the nearest obstacle. Often, sensors are used to derive this spatial information, and visual displays are deployed as interfaces to communicate this information to the operator. In this paper, we present psychoacoustic sonification as an alternative interface for human-machine interaction. Instead of visualizations, an interactive sound guides the operator to the desired target location, or helps her avoid obstacles in space. By considering psychoacoustics-i.e., the relationship between the physical and the perceptual attributes of sound-in the audio signal processing, we can communicate precisely and unambiguously interpretable direction and distance cues along three orthogonal axes to a user. We present exemplary use cases from various application areas where users can benefit from psychoacoustic sonification.
... In this section the psychoacoustic auditory display is introduced, followed by a description of the experiment. Details on the technical implementation [13,15], some previous experiments [13,16,17,18] and some deeper insights into the psychoacoustic considerations [13,15,18] can be found in the cited literature. ...
Conference Paper
An auditory display for short-range navigation in two-dimensional space is derived, implemented, and verified in an experiment. The auditory display design considers auditory qualities and auditory scene analysis principles. Orthogonal spatial dimensions are mapped to different auditory qualities in ranges at which they can be considered as orthogonal. This way, perceptual interactions are avoided and the auditory display is unambiguous. In particular, perceptual qualities - like perceived duration, loudness and loudness fluctuation, inharmonicity, roughness, noisiness, and pitch - and principles of simultaneous and sequential grouping, known from research in the field of auditory scene analysis, are considered. With the perceptual auditory display, 18 inexperienced users were able to find invisible targets on a computer screen. To quantify their performance, we calculated the length, entropy, box counting, target overshoots, motion interruptions, x- and y-flips of the mouse trajectories and measured the time needed to find the target. We furthermore identified six strategies, which the participants used to approach the target. The results show that they were able to combine the information of the two dimensions to derive the location of the target and approach it directly. But they were also able to find the target by approaching it dimension after dimension.
Article
A psychoacoustically motivated sonification design to guide clinicians in two-dimensional space is presented, e.g., to navigate a scalpel towards a target resection trajectory, ablation needle towards a pre-planned insertion point, or drill towards a target burrhole for craniotomy. Navigation is achieved by mapping spatial dimensions to audio synthesis parameters that affect the magnitude of different perceptual sound qualities. Orthogonal spatial dimensions are mapped to orthogonal auditory qualities.In a preliminary laboratory study, non-expert users successfully identified the target field out of 16 possible fields in 41% of all trials. The correct cardinal direction was identified in 84% of the trials. Based on both findings and further psychoacoustic considerations, the mapping range is optimized, and an implementation of an additional depth dimension is discussed.
Article
A psychoacoustically motivated sonification design is presented. It is intended to help surgeons navigate their scalpel to a target location in two-dimensional space. This is achieved by mapping spatial dimensions to audio parameters that affect the magnitude of different perceptual sound qualities. Distance and direction in the horizontal dimension are mapped to glissando speed and direction of a Shepard tone. The vertical dimension is divided into two regions. Below the target, the vertical distance controls the LFO speed of an amplitude modulation to create the impression of a regular beating well below the threshold of roughness sensation. Above the target height, the vertical deflection controls the depth of a frequency modulation to gradually increase the number and amplitudes of inharmonic partials, which affects the perceived consonance, noisiness and roughness. This redundancy is necessary because the magnitudes of each single sound quality are only differentiable with little confidence. In a preliminary study, non-surgeons successfully identified the target field out of 16 possible fields in 41% of all trials. The correct cardinal direction was identified in 84% of the trials. Based on findings and further psychoacoustic considerations, the mapping range is optimized and an implementation of an additional depth dimension is discussed.