PreprintPDF Available

Making Sense: Juxtaposing Visual, Auditory, and Kinesthetic Design Elements to Create Meaning, Reinforce Emotions, and Strengthen Player Memory Formation and Retrieval

Authors:
  • Mediadesign Hochschule für Design und Informatik
Preprints and early-stage research may not have been peer reviewed yet.

Abstract

In this paper, visual, auditory, and kinesthetic design elements are introduced with regard to their sense-specific narrative qualities or properties. Discussing perceived hierarchies of different sensory stimuli and their reciprocal effects through contrast and agreement, the paper shows how different forms of juxtaposition create meaning, reinforce player emotions, strengthen memory formation and retrieval.
Making Sense: Juxtaposing Visual, Auditory, and Kinesthetic !
Design Elements to Create Meaning, Reinforce Emotions, and !
Strengthen Player Memory Formation and Retrieval
J. Martin
Mediadesign University of Applied Sciences, Duesseldorf, Germany
j.martin@mediadesign.de
Abstract
In this paper, visual, auditory, and kinesthetic design elements are intro-
duced with regard to their sense-specific narrative qualities or properties.
Discussing perceived hierarchies of dierent sensory stimuli and their reci-
procal eects through contrast and agreement, the paper shows how dif-
ferent forms of juxtaposition create meaning, reinforce player emotions,
strengthen memory formation and retrieval, and support various dramatic
functions within individual beats through incongruity and ambiguity and
through clarification and intensification.
Keywords: game design, sensory design, narrativity
1. INTRODUCTION
Visual, auditory, and kinesthetic design elements in video games both inform the play-
er and elicit emotions. To establish a memorable gameplay moment in an individual
beat as the smallest element of structure [1], information and emotion design need to
go hand in hand. Moreover, memory retrieval is strengthened when the moment of
memory formation—i.e., the learning event—is attached to a specific context or space
that can then be recreated for easier retrieval [2][3][4]—which, as a caveat, works bet-
ter with recall than with recognition [5][6]. In turn, the specificity of a given context or
space is strengthened when it is attached to distinct emotions with the just-right level
of intensity [7].!
In dramatically complete games with story and character arcs, emotions are
primarily created through narrative content, or story. But emotions can also be created
through narrativity. Narrativity, a term adapted from film theory [8]—which, in this par-
ticular context, is closer in use to visual arts theory [9] than textual semiotics [10]—
refers to narrative qualities or properties that works of art often possess, in any medi-
um, despite lacking identifiable plot or story elements. Conveyed through visual, audi-
tory, and kinesthetic design elements, these narrative qualities or properties can sup-
port narrative content, aid memory formation and retrieval, and create memorable
gameplay moments beyond and without story or plot.!
! 2
J. Martin
This paper discusses the game design–specific characteristics of visual, audito-
ry, and kinesthetic design elements with regard to narrativity; how these experiences
relate to each other hierarchically; and how they can be combined to create ambiguity
and uncertainty on the one hand or work toward clarification and intensification on the
other to establish distinctly dierent meanings.!
2. SENSORY DESIGN ELEMENTS
Despite occasional attempts to add olfactory stimuli to media experiences [11][12][13],
smell and taste can be excluded as sensory design elements for video games at this
point in time. Visual and auditory design elements, in contrast, are both universally
applied and extensively researched, psychologically, neurologically, and aesthetically,
for audiovisual media and interactive media as well. Somatosensory design elements
for the last of the five traditional senses, the sense less formally known as touch,
needs a more thorough introduction.!
In the context of video game design, somatosensory design elements have been
studied and applied substantially less than visual or auditory ones. Moreover, the input
and output characteristics of video games work in ways that are not easily captured
by the term somatosensory, or somatosensory alone. Certain somatosensory cate-
gories are not productive, such as nociception [14] for the reception of harmful stimuli,
attached to pain responses [15], or haptic stimuli as the active exploration of surfaces
and objects [16]. Categories that are productive, on the other hand, work together with
the somatosensory system but are not necessarily an integral part of it, notably equi-
librioception [17] as the sense of balance. As a nontraditional sense, equilibrioception
combines visual and auditory information with proprioception [18], the sense of posi-
tion and movement—which, in turn, is part of the somatosensory palette.!
Thus, equilibrioception and proprioception appear to provide the most produc-
tive stimuli for game design purposes: equilibrioception on the one hand as the sense
of balance, acceleration (which includes both a sense of weight and eort), and direc-
tion of movement, and proprioception on the other as the sense of self-movement and
the relative positions and movements of body parts with respect to each other. As a
shorthand, and for the purposes of this paper, this combination will be referred to as
kinesthetic, an alternative term for proprioception that, especially in game design liter-
ature, often includes equilibrioceptive elements [19][20][21][22].!
2.1 Visual design elements
For visual content, narrative qualities or properties can be established through the
choice of color, color range, tint, shade, tone, surface texture, brightness, contrast,
exposure, hue, saturation, luminance, temperature, fluorescence, sharpness, haze,
blur, noise/grain, depth, resolution, size/dimensions, lighting, lenses, filters, camera
angles, subjective/objective view, and many more.!
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/
Making Sense
!3
2.2 Auditory design elements
For auditory content, primarily from music and Foley but from speech elements as
well, narrative qualities or properties can be established through duration, loudness,
timbre, pitch, intonation, modulation, inflection, rhythm, tempo, voice quality, modula-
tion eects like distortion, reverb, echo, and many others, as well as positional audio
in three-dimensional space and temporary silence. !
2.3 Kinesthetic design elements
For kinesthetic content, which strongly depends on a game’s control scheme and this
control scheme’s level of abstraction, narrative qualities or properties can be delivered
through combinations of player movement, avatar movement, and camera movement.
With regard to abstraction, three basic types can be dierentiated: traditional device-
specific controller setups like keyboard and mouse, gamepad, or touchscreen com-
mands; controller setups that mimic real-world tools like guns, steering wheels, musi-
cal instruments, rudder pedals, and similar; and motion sensor controller setups like
dance pads, the Wii Remote, or VR controllers. Sometimes these classes overlap, as
in the case of sensor-controlled clubs for screen-based and VR golf simulators, re-
spectively. But in terms of input abstraction, these classes structure the field su-
ciently well with regard to principal design decisions. Thus, while each type has its
own toolbox, narrative qualities or properties can be established through speed, ac-
celeration, force, angle, momentum, fluidity, sureness, smoothness, rhythm, balance,
economy, consistency, variety, predictability, unpredictability, and similar.!
2.4 Sensory design elements and emotions
While the open question of how these sensory design elements’ qualities or properties
evoke emotions in the player cannot be discussed at any depth in this paper, it needs
at least to be touched upon. Music can be highlighted as a representative example.
Despite its well-established research history, the question of how exactly musical ele-
ments—i.e., its qualities or properties—evoke emotions in listeners is far from solved,
let alone the question of how these emotions give rise to perceptions of narrativity [23]
[24][25]. Equally unresolved is the related question why many people are drawn to
pieces of music that evoke specific emotions like sadness, again attached to percep-
tions of narrativity [26]. These questions around emotions and narrativity for music
and, in extension, all other sensory stimuli as enumerated above, is a complex one,
owing to its broad range of implications. They span numerous fields from psychology
and neuropsychology to philosophy and art, the latter particularly through artistic pro-
cesses of abstraction in any medium that are able to encode creative intentions, at
least to a certain degree, and communicate an artistic vision!
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/
! 4
J. Martin
3. USE CASES FOR SENSORY DESIGN ELEMENTS
In most games, not counting certain types of experimental, serious, or art games, at
least two sensory design elements are present in any given beat, namely visual and
auditory ones. This presence also includes their absence in the form of temporary si-
lence or a temporary blank screen. This section will discuss how visual, auditory, and
kinesthetic design elements relate to each other hierarchically, as perceived by the
player, and how dierent sensory design elements can work together or against each
other to create specific eects and create dierent meanings.!
3.1 Juxtaposing sensory design elements for contrast
According to research, visual information dominates spatial processing while auditory
information dominates in other areas, notably temporal processing [27], emotion/
mood [28][29], and even empathic emotion [30]. Particularly with respect to emotions,
sound almost always overrides even strong visual expressions. Thus, creative and
imaginative design eects can be achieved in any given beat by juxtaposing and con-
trasting auditory and visual design elements for eects of incongruity, ambiguity, and
uncertainty. Strong kinesthetic design elements related to balance and movement can
certainly drown out both visual and auditory ones, but not necessarily in all three
classes as discussed above, and not across all dimensions of valence and arousal in
dimensional models of emotion [31]. The likelihood that kinesthetic design elements
can dominate visual and auditory ones decreases rapidly from motion sensor con-
troller setups to controller setups that mimic real-world tools to traditional device-spe-
cific controller setups like keyboard and mouse, gamepad, or touchscreen commands.
First and foremost, this is an eect of increasing abstraction, but the dierent types of
movement also play a role. During interactive sequences with traditional controller
setups, the avatar movement dominates all other movements. Player movement is
substantially reduced and camera movement, outside of cutscenes, is severely limit-
ed—kinesthetic design elements that rely on camera positioning and camera move-
ment work well in audiovisual media like movies, but do not work well together with
player agency in interactive environments like video games. Nevertheless, kinesthetic
design elements can also be juxtaposed in contrasting ways with visual or auditory
ones, or both, to create interesting and unexpected eects around incongruence, am-
biguity, and uncertainty.!
3.2 Juxtaposing sensory design elements for agreement
Barring certain forms of movement design in kinesthetic setups, auditory design ele-
ments are again the strongest factor in most cases when visual, auditory, and kines-
thetic design elements work together in agreement. The two most notable eects that
can be achieved through the agreement of sensory design elements is clarification
and intensification. To exemplify their creative potential for game design, they will be
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/
Making Sense
!5
discussed in this section in association with the Kuleshov montage eect and the mis-
attribution of arousal eect, respectively.!
Compared to defined narratives and story beats, narrative qualities or properties
of sensory design elements leave substantially more room for interpretation, and there
will always remain uncertainty with regard to this interpretation. To support a preferred
interpretation through design, one sensory design element can clarify another sensory
design element by, e.g., utilizing the Kuleshov montage eect [32][33]. As Kuleshov
showed, people take cues for the interpretation of a picture from a second picture that
is juxtaposed with it; and when that second picture is dierent, the first picture will be
interpreted dierently. That way, the second picture defines or clarifies the meaning of
the first picture by providing cues for interpretation, which can then resolve ambigui-
ties and create specific meaning. As a more recent study on a potential auditory
Kuleshov eect showed [34], juxtaposing a visual design element with dierent audito-
ry design elements yields similar results—depending on the nature of the juxtaposed
auditory design element, the visual one will be interpreted dierently, its ambiguities
will be resolved in dierent ways, and the meaning that is created will also be dierent.
Thus, juxtaposing dierent sensory design elements in agreement with each other can
help the player understand a situation, e.g., assessing the trustworthiness of a non-
player character or estimating the relative importance of an item, without resorting to
explanation or exposition.!
In contrast to clarification, intensification seems a more obvious choice, and it is
perhaps the most frequently applied eect in this context. Yet, narrative qualities or
properties from dierent sensory design elements working in agreement can intensify
player emotions very precisely and very eectively in less obvious ways. The misattri-
bution of arousal eect has been attested to by several studies, the most widely
known of which is the “bridge” experiment [35]. In this setup, an attractive “interview-
er” meets the test subjects on a suspension bridge with “many arousal-inducing fea-
tures,” while the subjects from the control group meet the same interviewer on a solid
wood bridge much closer to the ground. As a reproducible eect, the test subjects
from the suspension bridge are significantly more attracted to the interviewer than the
subjects from the control group. This result was reinforced by other, methodically more
rigorous (but less spectacular) tests within the same study. Moreover, other studies
showed that this eect of misattributed arousal works not only with emotions from the
fear spectrum, as in the bridge experiment, but also with emotions like euphoria and
anger [36][37] or confidence [38], and the emotional transfer even works with physical
exertion [39][40] and high-arousal music [41]. Dependent on the setup, these transfer
eects can intensify both positive attraction and its opposite, negative attraction. That
way, narrative qualities or properties from visual, auditory, and kinesthetic design ele-
ments that work in agreement can intensify an intended player emotion and thereby
strengthen the player’s relationship with non-player characters, places, or items in
positive as well as in negative ways.!
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/
! 6
J. Martin
3.3 Misleading sensory design elements
A question that arises, specifically with clarification in mind, is whether narrative quali-
ties or properties from sensory design elements should be used to mislead the player.
Except, again, for certain kinds of experimental, serious, or art games, the general an-
swer is that diegetic sensory design elements are allowed to mislead but non-diegetic
ones are not. From the design perspective, a cheerful sing-along within the game
world is certainly allowed to deceive the player as to its participants’ intentions. But
the same tune introducing a dangerous situation as non-diegetic music from outside
the game world, that would be dicult to justify.!
4. CONCLUSION
With respect to the player, the perceived hierarchies and the use of contrast and
agreement strategies for narrative qualities or properties attached to visual, auditory,
and kinesthetic design elements can be used to create disorienting experiences like
incongruity, ambiguity, or uncertainty; create meaning and understanding through clar-
ification; reinforce emotional bonds through intensification; and strengthen memory
formation and retrieval. While narrative qualities or properties cannot be used to ad-
vance the plot, as they have no identifiable story elements by definition, they can still
be used to enrich plot points with specific emotional values. Beyond that, these narra-
tive qualities or properties are also able to support many other dramatic functions
within individual beats, e.g., portray a character, communicate an insight into the
game world, or advance player proficiency with regard to knowledge and understand-
ing.!
REFERENCES
[1]$McKee, Robert. Story: Substance, Structure, Style, and the Principles of Screenwriting.
New York: ReganBooks, 1997. 37.!
[2]$Godden, Duncan & Alan Baddeley. “When Does Context Influence Recognition
Memory?” British Journal of Psychology Vol.71 (1), 1980. 99–104.!
[3]$Smith, Stephen M. “Remembering in and Out of Context.” Journal of Experimental
Psychology: Human Learning and Memory Vol.5 (5), 1979. 460–71. !
[4]$Smith, Stephen M., Arthur Glenberg, & Robert A. Bjork. “Environmental Context and
Human Memory.” Memory & Cognition Vol.6 (4), 1978. 342–53.!
[5]$Isarida, Takeo, Toshiko K. Isarida, & Tetsuya Sakai. “Eects of Study Time and
Meaningfulness on Environmental Context-Dependent Recognition.” Memory & Cogni-
tion Vol.40 (8), 2012. 1225–35.!
[6]$Smith, Stephen M. & Edward Vela. “Environmental Context-Dependent Memory: A
Review and Meta-Analysis.” Psychonomic Bulletin & Review Vol.8 (2), 2001. 203–20.!
[7]$Ack Baraly, Kylee T., Pascal Hot, Patrick S. R. Davidson, & Deborah Talmi. “How
Emotional Arousal Enhances Episodic Memory.” In: Learning and Memory: A Compre-
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/
Making Sense
!7
hensive Reference. Vol.2: Cognitive Psychology of Memory. Vol. Ed. John T. Wixted. Se-
ries Ed. John H. Byrne. 2nd Ed. Oxford: Academic Press, 2017. 295–324.!
[8]$Scholes, Robert. “Narration and Narrativity in Film.” Quarterly Review of Film Studies Vol.
1 (3), 1976. 283–96.!
[9]$Wolf, Werne. “Narrative and Narrativity: A Narratological Reconceptualization and its
Applicability to the Visual Arts.” Word & Image: A Journal of Verbal/Visual Enquiry, Vol.19
(3), 180–97.!
[10]$Ryan, Marie-Laure. “The Modes of Narrativity and Their Visual Metaphors.” Style Vol.26
(3), 1992. 368-87.!
[11]$Paterson, M. W. D. “Digital Scratch and Virtual Sni.” In: The Smell Culture Reader. Ed.
Jim Drobnick. New York: Berg, 2006. 358–70.!
[12]$Gilbert, Avery. What the Nose Knows: The Science of Scent in Everyday Life. New York:
Crown, 2008. 147–69.!
[13]$Olofsson, Jonas K., Simon Niedenthal, Marie Ehrndal, Marta Zakrzewska, Andreas
Wartel, & Maria Larsson. “Beyond Smell-O-Vision: Possibilities for Smell-Based Digital
Media.” Simulation & Gaming Vol.48 (4), 2017. 455–79. !
[14 ]$Sherrington, Charles. S. The Integrative Action of the Nervous System. Yale University
Mrs. Hepsa Ely Silliman Memorial Lectures. New Haven: Yale UP, 1906. 226.!
[15]$Smith, Ewan St. John. “Advances in Understanding Nociception and Neuropathic Pain.”
Journal of Neurology Vol.265 (2), 2018. 231–38.!
[16]$Ledermann, Susan J. & Roberta L. Klatzky. “Haptic Perception: A Tutorial.” Attention,
Perception, & Psychophysics Vol.71 (7), 2009. 1439–59.!
[17]$Atwater, Riannon C. “Exteroception.” In: The Five Senses and Beyond. The Encyclopedia
of Perception. Ed. Jennifer L. Hellier. Santa Barbara: Greenwood, 2017. 152–55. 154–55.!
[18]$Tuthill, John C. & Eiman Azim. “Proprioception.” Current Biology Vol.28 (5), 2018. R194–
R203.!
[19]$Aukstakalnis, Steve. Practical Augmented Reality: A Guide to the Technologies,
Applications, and Human Factors for AR and VR. Boston: Addison-Wesley, 2017.!
[20]$Blade, Richard A. & Mary Lou Padgett. “Virtual Environments Standards and Ter-
minology.” In: Handbook of Virtual Environments: Design, Implementation, and Ap-
plications. Ed. Kelly S. Hale & Kay M. Stanney. Boca Raton: CRC, 20152. 23–35.!
[21]$Calleja, Gordon, Laura Herrewijn, & Karolien Poels. “Aective Involvement in Digital
Games.” In: Emotion in Games: Theory and Praxis. Ed. Kostas Karpouzis & Georgios N.
Yannakakis. Springer, 2016. 39–56.!
[22]$Swink, Steve. Game Feel: A Game Designer's Guide to Virtual Sensation. Burlington:
Morgan Kaufmann, 2004.!
[23]$Gomez, Patrick & Brigitta Danuser. “Relationships Between Musical Structure and
Psychophysiological Measures of Emotion.” Emotion Vol.7 (2), 2007. 377–87.!
[24]$Robinson, Jenefer. Deeper than Reason: Emotion and its Role in Literature, Music, and
Art. Oxford: Oxford UP, 2005. 310–13.!
[25]$Tarasti, Eero. Semiotics of Classical Music: How Mozart, Brahms, and Wagner Talk to Us.
Berlin: de Gruyter, 2012.!
[26]$Garrido, Sandra. Why Are We Attracted to Sad Music? London: Palgrave Macmillan,
2017.!
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/
! 8
J. Martin
[27]!Sharon E. Guttman, Lee A. Gilroy, & Randolph Blake. “Hearing What the Eyes See:
Auditory Encoding of Visual Temporal Sequences.” Psychological Science Vol.16 (3),
2005. 228–35."
[28] !Jacob Jolij & Maaike Meurs. “Music Alters Visual Perception.” PLoS ONE Vol.6 (4):
e18861, 2011. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.
0018861"
[29]!Zadra, Jonathan R. & Gerald Clore. “Emotion and Perception: The Role of Aective
Information.” Wiley Interdisciplinary Reviews. Cognitive Science Vol.2 (6). 676–85."
[30]!Kraus, Michael W. “Voice-Only Communication Enhances Empathic Accuracy.”
American Psychologist Vol.72 (7), 2017. 644–54."
[31]!Rubin, David C. & Jennifer M. Talarico. “A Comparison of Dimensional Models of
Emotion". Memory Vol.17 (8), 2009. 802–08."
[32]!Mobbs, Dean, Nikolaus Weiskopf, Hakwan C. Lau, Eric Featherstone, Ray J. Dolan, &
Chris D. Frith. “The Kuleshov Eect: The Influence of Contextual Framing on Emotional
Attributions.” Social Cognitive and Aective Neuroscience Vol.1 (2), 2006. 95–106."
[33]!Barratt, Daniel, Anna Cabak Rédei, Åse Innes-Ker, & Joost van de Weijer. “Does the
Kuleshov Eect Really Exist? Revisiting a Classic Film Experiment on Facial Expressions
and Emotional Contexts.” Perception Vol.45 (8), 2016. 847–74."
[34]!Baranowski, Andreas M., & Heiko Hecht. “The Auditory Kuleshov Eect: Multisensory
Integration in Movie Editing.” Perception Vol 46 (5), 2017. 624–31."
[35]!Dutton, Donald G. & Arthur P. Aron. “Some Evidence for Heightened Sexual Attraction
Under Conditions of High Anxiety.” Journal of Personality and Social Psychology 30 (4),
1974. 510–17."
[36]!Schachter, Stanley & Jerome Singer. “Cognitive, Social, and Physiological Determinants
of Emotional State". Psychological Review Vol.69 (5), 1962. 379–99."
[37]!Cotton, John. “A Review of Research on Schachter’s Theory of Emotion and the
Misattribution of Arousal.” European Journal of Social Psychology Vol.11 (4), 1981. 365–
97."
[38]!Savitsky, Kenneth, Victoria Husted Medvec, Ann E. Charlton, & Thomas Gilovich. “‘What,
Me Worry?’: Arousal, Misattribution, and the Eect of Temporal Distance on
Confidence.” Personality and Social Psychology Bulletin Vol.24 (5), 1998. 529–36."
[39]!White, Gregory L., Sanford Fishbein, & Jerey Rutsein. “Passionate Love and the
Misattribution of Arousal.” Social Psychology Vol.41 (1), 1981. 56–62."
[40]!Allen, James B., Douglas T. Kenrick, Darwyn E. Linder, Michael A. McCall. “Arousal and
Attraction: A Response-Facilitation Alternative to Misattribution and Negative-Rein-
forcement Models.” Journal of Personality and Social Psychology Vol.57 (2), 1989. 261–
70."
[41]!Marin, Manuela M., Raphaela Schober, Bruno Gingras, & Helmut Leder. “Misattribution
of Musical Arousal Increases Sexual Attraction Towards Opposite-Sex Faces in
Females.” PLoS One Vol.12 (9): e0183531, 2017. https://www.ncbi.nlm.nih.gov/pmc/arti-
cles/PMC5593195/ "
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
Pain results from the activation of a subset of sensory neurones termed nociceptors and has evolved as a “detect and protect” mechanism. However, lesion or disease in the sensory system can result in neuropathic pain, which serves no protective function. Understanding how the sensory nervous system works and what changes occur in neuropathic pain are vital in identifying new therapeutic targets and developing novel analgesics. In recent years, technologies such as optogenetics and RNA-sequencing have been developed, which alongside the more traditional use of animal neuropathic pain models and insights from genetic variations in humans have enabled significant advances to be made in the mechanistic understanding of neuropathic pain.
Article
Full-text available
This research tests the prediction that voice-only communication increases empathic accuracy over communication across senses. We theorized that people often intentionally communicate their feelings and internal states through the voice, and as such, voice-only communication allows perceivers to focus their attention on the channel of communication most active and accurate in conveying emotions to others. We used 5 experiments to test this hypothesis (N = 1,772), finding that voice-only communication elicits higher rates of empathic accuracy relative to vision-only and multisense communication both while engaging in interactions and perceiving emotions in recorded interactions of strangers. Experiments 4 and 5 reveal that voice-only communication is particularly likely to enhance empathic accuracy through increasing focused attention on the linguistic and paralinguistic vocal cues that accompany speech. Overall, the studies question the primary role of the face in communication of emotion, and offer new insights for improving emotion recognition accuracy in social interactions.
Article
Full-text available
Several theories about the origins of music have emphasized its biological and social functions, including in courtship. Music may act as a courtship display due to its capacity to vary in complexity and emotional content. Support for music’s reproductive function comes from the recent finding that only women in the fertile phase of the reproductive cycle prefer composers of complex melodies to composers of simple ones as short-term sexual partners, which is also in line with the ovulatory shift hypothesis. However, the precise mechanisms by which music may influence sexual attraction are unknown, specifically how music may interact with visual attractiveness cues and affect perception and behaviour in both genders. Using a crossmodal priming paradigm, we examined whether listening to music influences ratings of facial attractiveness and dating desirability of opposite-sex faces. We also tested whether misattribution of arousal or pleasantness underlies these effects, and explored whether sex differences and menstrual cycle phase may be moderators. Our sample comprised 64 women in the fertile or infertile phase (no hormonal contraception use) and 32 men, carefully matched for mood, relationship status, and musical preferences. Musical primes (25 s) varied in arousal and pleasantness, and targets were photos of faces with neutral expressions (2 s). Group-wise analyses indicated that women, but not men, gave significantly higher ratings of facial attractiveness and dating desirability after having listened to music than in the silent control condition. High-arousing, complex music yielded the largest effects, suggesting that music may affect human courtship behaviour through induced arousal, which calls for further studies on the mechanisms by which music affects sexual attraction in real-life social contexts.
Article
Full-text available
Research Problem: The purpose of this research synthesis is to identify new opportunities for smell-enabled games based upon current olfactory research, and to present early game concepts that have emerged from our empirical assessments. Literature Review: We briefly summarize key projects in the history of scent technologies for film and media. Human-Computer Interaction researchers have also explored a number of uses for scent delivery in interactive digital media. Recent developments in olfactory psychology and neuroscience research suggest that a fruitful avenue for exploration is to develop learning games that expand olfactory capacity. Methodology: We have conducted two studies of computer-based perceptual and cognitive olfactory tasks. Mixture perception experiment: We designed a perceptual experiment where the task was to correctly estimate the intensity of odor components in a blend of coffee and tea. Blended odors were presented to 10 healthy adults by means of a computer-controlled olfactometer. Following each stimulation, the participant used a computer interface to estimate the intensity of components of the blend. Event-based memory experiment: We have developed a digital olfactory version of the children’s game “Memory.” The game interface consists of 32 white squares that are presented in a grid pattern on the screen and that, when participants click on them, triggers the release of one of eight possible smells from the olfactometer. Fifteen healthy adult participants were tested in 10 laboratory sessions distributed over three weeks. Results and Conclusions: Our empirical results suggest that smell training through learning games holds promise as a means of improving cognitive function. The results of our event-based memory experiment suggest that both olfactory and visual memory capacities might have benefitted from olfactory game training. The results of our mixture perception experiment indicate that binary odor mixtures might provide a suitable starting point for perceptual training, and we suggest that a smell-enabled game might include adaptive difficulty by progressively introducing more complex mixtures. We have used event-based memory and mixture perception as “olfactory targets” for game mechanic development, and present early design concepts for “Smelly Genes” and “Scenter.” Finally, we discuss future directions and challenges for this new, interdisciplinary research topic.
Book
Aims and Scope Musical semiotics is a new discipline and paradigm of both semiotics and musicology. In its tradition, the current volume constitutes a radically new solution to the theoretical problem of how musical meanings emerge and how they are transmitted by musical signs even in most "absolute" and abstract musical works of Western classical heritage. Works from symphonies, lied, chamber music to opera are approached and studied here with methods of semiotic inspiration. Its analyses stem from systematic methods in the author's previous work, yet totally new analytic concepts are also launched in order to elucidate profound musical significations verbally. The book reflects the new phase in the author's semiotic approach, the one characterized by the so-called "existential semiotics" elaborated on the basis of philosophers from Kant , Hegel and Kierkegaard to Jaspers, Heidegger, Sartre and Marcel. The key notions like musical subject, Schein, becoming, temporality, modalities, Dasein, transcendence put musical facts in a completely new light and perspectives of interpretation. The volume attempts to make explicit what is implicit in every musical interpretation, intuition and understanding: to explain how compositions and composers "talk" to us. Its analyses are accessible due to the book's universal approach. Music is experienced as a language, communicating from one subject to another. © 2012 Walter de Gruyter GmbH & Co. KG, Berlin/Boston. All Rights Reserved.
Article
In this introductory chapter, Garrido looks at the popularity of listening to sad music both historically and in the twenty-first century, and the philosophical paradox that an attraction to sad music presents. With a focus on key minds in history such as Aristotle and Dr. Robert Burton, Garrido introduces an argument that is central to the rest of the book: the need to consider individual differences when looking at the question of why we listen to sad music.
Article
Almost a hundred years ago, the Russian filmmaker Lev Kuleshov conducted his now famous editing experiment in which different objects were added to a given film scene featuring a neutral face. It is said that the audience interpreted the unchanged facial expression as a function of the added object (e.g., an added soup made the face express hunger). This interaction effect has been dubbed “Kuleshov effect.” In the current study, we explored the role of sound in the evaluation of facial expressions in films. Thirty participants watched different clips of faces that were intercut with neutral scenes, featuring either happy music, sad music, or no music at all. This was crossed with the facial expressions of happy, sad, or neutral. We found that the music significantly influenced participants’ emotional judgments of facial expression. Thus, the intersensory effects of music are more specific than previously thought. They alter the evaluation of film scenes and can give meaning to ambiguous situations.
Chapter
The chapter takes as its main object of study (Calleja’s (2011) In-game: from immersion to incorporation. MIT Press, Cambridge, MA) Player Involvement Model, an analytical framework designed to understand player experience developed through qualitative research. The model identifies six dimensions of involvement in digital games, namely kinesthetic involvement, spatial involvement, shared involvement, narrative involvement, ludic involvement and affective involvement. The goal of the chapter was to develop the model further by testing it in an experimental context. Consequently, three experiments were conducted in order to examine how different components of digital gameplay (i.e. the story that is written into a game, the social setting in which a game is played, and the player’s control in a game environment) can affect the player’s involvement on the six proposed dimensions. Special attention is paid to affective involvement, and how this dimension of player involvement relates to the other dimensions. The findings of the experiments provide initial support for (Calleja’s (2011) In-game: from immersion to incorporation. MIT Press, Cambridge, MA) Player Involvement Model in a quantitative setting.
Article
According to film mythology, the Soviet filmmaker Lev Kuleshov conducted an experiment in which he combined a close-up of an actor's neutral face with three different emotional contexts: happiness, sadness, and hunger. The viewers of the three film sequences reportedly perceived the actor's face as expressing an emotion congruent with the given context. It is not clear, however, whether or not the so-called "Kuleshov effect" really exists. The original film footage is lost and recent attempts at replication have produced either conflicting or unreliable results. The current paper describes an attempt to replicate Kuleshov's original experiment using an improved experimental design. In a behavioral and eye tracking study, 36 participants were each presented with 24 film sequences of neutral faces across six emotional conditions. For each film sequence, the participants were asked to evaluate the emotion of the target person in terms of valence, arousal, and category. The participants' eye movements were recorded throughout. The results suggest that some sort of Kuleshov effect does in fact exist. For each emotional condition, the participants tended to choose the appropriate category more frequently than the alternative options, while the answers to the valence and arousal questions also went in the expected directions. The eye tracking data showed how the participants attended to different regions of the target person's face (in light of the intermediate context), but did not reveal the expected differences between the emotional conditions.