Content uploaded by J. Martin
Author content
All content in this area was uploaded by J. Martin on Jul 29, 2019
Content may be subject to copyright.
Making Sense: Juxtaposing Visual, Auditory, and Kinesthetic !
Design Elements to Create Meaning, Reinforce Emotions, and !
Strengthen Player Memory Formation and Retrieval
J. Martin
Mediadesign University of Applied Sciences, Duesseldorf, Germany
j.martin@mediadesign.de
Abstract
In this paper, visual, auditory, and kinesthetic design elements are intro-
duced with regard to their sense-specific narrative qualities or properties.
Discussing perceived hierarchies of different sensory stimuli and their reci-
procal effects through contrast and agreement, the paper shows how dif-
ferent forms of juxtaposition create meaning, reinforce player emotions,
strengthen memory formation and retrieval, and support various dramatic
functions within individual beats through incongruity and ambiguity and
through clarification and intensification.
Keywords: game design, sensory design, narrativity
1. INTRODUCTION
Visual, auditory, and kinesthetic design elements in video games both inform the play-
er and elicit emotions. To establish a memorable gameplay moment in an individual
beat as the smallest element of structure [1], information and emotion design need to
go hand in hand. Moreover, memory retrieval is strengthened when the moment of
memory formation—i.e., the learning event—is attached to a specific context or space
that can then be recreated for easier retrieval [2][3][4]—which, as a caveat, works bet-
ter with recall than with recognition [5][6]. In turn, the specificity of a given context or
space is strengthened when it is attached to distinct emotions with the just-right level
of intensity [7].!
In dramatically complete games with story and character arcs, emotions are
primarily created through narrative content, or story. But emotions can also be created
through narrativity. Narrativity, a term adapted from film theory [8]—which, in this par-
ticular context, is closer in use to visual arts theory [9] than textual semiotics [10]—
refers to narrative qualities or properties that works of art often possess, in any medi-
um, despite lacking identifiable plot or story elements. Conveyed through visual, audi-
tory, and kinesthetic design elements, these narrative qualities or properties can sup-
port narrative content, aid memory formation and retrieval, and create memorable
gameplay moments beyond and without story or plot.!
! 2
J. Martin
This paper discusses the game design–specific characteristics of visual, audito-
ry, and kinesthetic design elements with regard to narrativity; how these experiences
relate to each other hierarchically; and how they can be combined to create ambiguity
and uncertainty on the one hand or work toward clarification and intensification on the
other to establish distinctly different meanings.!
2. SENSORY DESIGN ELEMENTS
Despite occasional attempts to add olfactory stimuli to media experiences [11][12][13],
smell and taste can be excluded as sensory design elements for video games at this
point in time. Visual and auditory design elements, in contrast, are both universally
applied and extensively researched, psychologically, neurologically, and aesthetically,
for audiovisual media and interactive media as well. Somatosensory design elements
for the last of the five traditional senses, the sense less formally known as touch,
needs a more thorough introduction.!
In the context of video game design, somatosensory design elements have been
studied and applied substantially less than visual or auditory ones. Moreover, the input
and output characteristics of video games work in ways that are not easily captured
by the term somatosensory, or somatosensory alone. Certain somatosensory cate-
gories are not productive, such as nociception [14] for the reception of harmful stimuli,
attached to pain responses [15], or haptic stimuli as the active exploration of surfaces
and objects [16]. Categories that are productive, on the other hand, work together with
the somatosensory system but are not necessarily an integral part of it, notably equi-
librioception [17] as the sense of balance. As a nontraditional sense, equilibrioception
combines visual and auditory information with proprioception [18], the sense of posi-
tion and movement—which, in turn, is part of the somatosensory palette.!
Thus, equilibrioception and proprioception appear to provide the most produc-
tive stimuli for game design purposes: equilibrioception on the one hand as the sense
of balance, acceleration (which includes both a sense of weight and effort), and direc-
tion of movement, and proprioception on the other as the sense of self-movement and
the relative positions and movements of body parts with respect to each other. As a
shorthand, and for the purposes of this paper, this combination will be referred to as
kinesthetic, an alternative term for proprioception that, especially in game design liter-
ature, often includes equilibrioceptive elements [19][20][21][22].!
2.1 Visual design elements
For visual content, narrative qualities or properties can be established through the
choice of color, color range, tint, shade, tone, surface texture, brightness, contrast,
exposure, hue, saturation, luminance, temperature, fluorescence, sharpness, haze,
blur, noise/grain, depth, resolution, size/dimensions, lighting, lenses, filters, camera
angles, subjective/objective view, and many more.!
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/
Making Sense
!3
2.2 Auditory design elements
For auditory content, primarily from music and Foley but from speech elements as
well, narrative qualities or properties can be established through duration, loudness,
timbre, pitch, intonation, modulation, inflection, rhythm, tempo, voice quality, modula-
tion effects like distortion, reverb, echo, and many others, as well as positional audio
in three-dimensional space and temporary silence. !
2.3 Kinesthetic design elements
For kinesthetic content, which strongly depends on a game’s control scheme and this
control scheme’s level of abstraction, narrative qualities or properties can be delivered
through combinations of player movement, avatar movement, and camera movement.
With regard to abstraction, three basic types can be differentiated: traditional device-
specific controller setups like keyboard and mouse, gamepad, or touchscreen com-
mands; controller setups that mimic real-world tools like guns, steering wheels, musi-
cal instruments, rudder pedals, and similar; and motion sensor controller setups like
dance pads, the Wii Remote, or VR controllers. Sometimes these classes overlap, as
in the case of sensor-controlled clubs for screen-based and VR golf simulators, re-
spectively. But in terms of input abstraction, these classes structure the field suffi-
ciently well with regard to principal design decisions. Thus, while each type has its
own toolbox, narrative qualities or properties can be established through speed, ac-
celeration, force, angle, momentum, fluidity, sureness, smoothness, rhythm, balance,
economy, consistency, variety, predictability, unpredictability, and similar.!
2.4 Sensory design elements and emotions
While the open question of how these sensory design elements’ qualities or properties
evoke emotions in the player cannot be discussed at any depth in this paper, it needs
at least to be touched upon. Music can be highlighted as a representative example.
Despite its well-established research history, the question of how exactly musical ele-
ments—i.e., its qualities or properties—evoke emotions in listeners is far from solved,
let alone the question of how these emotions give rise to perceptions of narrativity [23]
[24][25]. Equally unresolved is the related question why many people are drawn to
pieces of music that evoke specific emotions like sadness, again attached to percep-
tions of narrativity [26]. These questions around emotions and narrativity for music
and, in extension, all other sensory stimuli as enumerated above, is a complex one,
owing to its broad range of implications. They span numerous fields from psychology
and neuropsychology to philosophy and art, the latter particularly through artistic pro-
cesses of abstraction in any medium that are able to encode creative intentions, at
least to a certain degree, and communicate an artistic vision!
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/
! 4
J. Martin
3. USE CASES FOR SENSORY DESIGN ELEMENTS
In most games, not counting certain types of experimental, serious, or art games, at
least two sensory design elements are present in any given beat, namely visual and
auditory ones. This presence also includes their absence in the form of temporary si-
lence or a temporary blank screen. This section will discuss how visual, auditory, and
kinesthetic design elements relate to each other hierarchically, as perceived by the
player, and how different sensory design elements can work together or against each
other to create specific effects and create different meanings.!
3.1 Juxtaposing sensory design elements for contrast
According to research, visual information dominates spatial processing while auditory
information dominates in other areas, notably temporal processing [27], emotion/
mood [28][29], and even empathic emotion [30]. Particularly with respect to emotions,
sound almost always overrides even strong visual expressions. Thus, creative and
imaginative design effects can be achieved in any given beat by juxtaposing and con-
trasting auditory and visual design elements for effects of incongruity, ambiguity, and
uncertainty. Strong kinesthetic design elements related to balance and movement can
certainly drown out both visual and auditory ones, but not necessarily in all three
classes as discussed above, and not across all dimensions of valence and arousal in
dimensional models of emotion [31]. The likelihood that kinesthetic design elements
can dominate visual and auditory ones decreases rapidly from motion sensor con-
troller setups to controller setups that mimic real-world tools to traditional device-spe-
cific controller setups like keyboard and mouse, gamepad, or touchscreen commands.
First and foremost, this is an effect of increasing abstraction, but the different types of
movement also play a role. During interactive sequences with traditional controller
setups, the avatar movement dominates all other movements. Player movement is
substantially reduced and camera movement, outside of cutscenes, is severely limit-
ed—kinesthetic design elements that rely on camera positioning and camera move-
ment work well in audiovisual media like movies, but do not work well together with
player agency in interactive environments like video games. Nevertheless, kinesthetic
design elements can also be juxtaposed in contrasting ways with visual or auditory
ones, or both, to create interesting and unexpected effects around incongruence, am-
biguity, and uncertainty.!
3.2 Juxtaposing sensory design elements for agreement
Barring certain forms of movement design in kinesthetic setups, auditory design ele-
ments are again the strongest factor in most cases when visual, auditory, and kines-
thetic design elements work together in agreement. The two most notable effects that
can be achieved through the agreement of sensory design elements is clarification
and intensification. To exemplify their creative potential for game design, they will be
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/
Making Sense
!5
discussed in this section in association with the Kuleshov montage effect and the mis-
attribution of arousal effect, respectively.!
Compared to defined narratives and story beats, narrative qualities or properties
of sensory design elements leave substantially more room for interpretation, and there
will always remain uncertainty with regard to this interpretation. To support a preferred
interpretation through design, one sensory design element can clarify another sensory
design element by, e.g., utilizing the Kuleshov montage effect [32][33]. As Kuleshov
showed, people take cues for the interpretation of a picture from a second picture that
is juxtaposed with it; and when that second picture is different, the first picture will be
interpreted differently. That way, the second picture defines or clarifies the meaning of
the first picture by providing cues for interpretation, which can then resolve ambigui-
ties and create specific meaning. As a more recent study on a potential auditory
Kuleshov effect showed [34], juxtaposing a visual design element with different audito-
ry design elements yields similar results—depending on the nature of the juxtaposed
auditory design element, the visual one will be interpreted differently, its ambiguities
will be resolved in different ways, and the meaning that is created will also be different.
Thus, juxtaposing different sensory design elements in agreement with each other can
help the player understand a situation, e.g., assessing the trustworthiness of a non-
player character or estimating the relative importance of an item, without resorting to
explanation or exposition.!
In contrast to clarification, intensification seems a more obvious choice, and it is
perhaps the most frequently applied effect in this context. Yet, narrative qualities or
properties from different sensory design elements working in agreement can intensify
player emotions very precisely and very effectively in less obvious ways. The misattri-
bution of arousal effect has been attested to by several studies, the most widely
known of which is the “bridge” experiment [35]. In this setup, an attractive “interview-
er” meets the test subjects on a suspension bridge with “many arousal-inducing fea-
tures,” while the subjects from the control group meet the same interviewer on a solid
wood bridge much closer to the ground. As a reproducible effect, the test subjects
from the suspension bridge are significantly more attracted to the interviewer than the
subjects from the control group. This result was reinforced by other, methodically more
rigorous (but less spectacular) tests within the same study. Moreover, other studies
showed that this effect of misattributed arousal works not only with emotions from the
fear spectrum, as in the bridge experiment, but also with emotions like euphoria and
anger [36][37] or confidence [38], and the emotional transfer even works with physical
exertion [39][40] and high-arousal music [41]. Dependent on the setup, these transfer
effects can intensify both positive attraction and its opposite, negative attraction. That
way, narrative qualities or properties from visual, auditory, and kinesthetic design ele-
ments that work in agreement can intensify an intended player emotion and thereby
strengthen the player’s relationship with non-player characters, places, or items in
positive as well as in negative ways.!
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/
! 6
J. Martin
3.3 Misleading sensory design elements
A question that arises, specifically with clarification in mind, is whether narrative quali-
ties or properties from sensory design elements should be used to mislead the player.
Except, again, for certain kinds of experimental, serious, or art games, the general an-
swer is that diegetic sensory design elements are allowed to mislead but non-diegetic
ones are not. From the design perspective, a cheerful sing-along within the game
world is certainly allowed to deceive the player as to its participants’ intentions. But
the same tune introducing a dangerous situation as non-diegetic music from outside
the game world, that would be difficult to justify.!
4. CONCLUSION
With respect to the player, the perceived hierarchies and the use of contrast and
agreement strategies for narrative qualities or properties attached to visual, auditory,
and kinesthetic design elements can be used to create disorienting experiences like
incongruity, ambiguity, or uncertainty; create meaning and understanding through clar-
ification; reinforce emotional bonds through intensification; and strengthen memory
formation and retrieval. While narrative qualities or properties cannot be used to ad-
vance the plot, as they have no identifiable story elements by definition, they can still
be used to enrich plot points with specific emotional values. Beyond that, these narra-
tive qualities or properties are also able to support many other dramatic functions
within individual beats, e.g., portray a character, communicate an insight into the
game world, or advance player proficiency with regard to knowledge and understand-
ing.!
REFERENCES
[1]$McKee, Robert. Story: Substance, Structure, Style, and the Principles of Screenwriting.
New York: ReganBooks, 1997. 37.!
[2]$Godden, Duncan & Alan Baddeley. “When Does Context Influence Recognition
Memory?” British Journal of Psychology Vol.71 (1), 1980. 99–104.!
[3]$Smith, Stephen M. “Remembering in and Out of Context.” Journal of Experimental
Psychology: Human Learning and Memory Vol.5 (5), 1979. 460–71. !
[4]$Smith, Stephen M., Arthur Glenberg, & Robert A. Bjork. “Environmental Context and
Human Memory.” Memory & Cognition Vol.6 (4), 1978. 342–53.!
[5]$Isarida, Takeo, Toshiko K. Isarida, & Tetsuya Sakai. “Effects of Study Time and
Meaningfulness on Environmental Context-Dependent Recognition.” Memory & Cogni-
tion Vol.40 (8), 2012. 1225–35.!
[6]$Smith, Stephen M. & Edward Vela. “Environmental Context-Dependent Memory: A
Review and Meta-Analysis.” Psychonomic Bulletin & Review Vol.8 (2), 2001. 203–20.!
[7]$Ack Baraly, Kylee T., Pascal Hot, Patrick S. R. Davidson, & Deborah Talmi. “How
Emotional Arousal Enhances Episodic Memory.” In: Learning and Memory: A Compre-
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/
Making Sense
!7
hensive Reference. Vol.2: Cognitive Psychology of Memory. Vol. Ed. John T. Wixted. Se-
ries Ed. John H. Byrne. 2nd Ed. Oxford: Academic Press, 2017. 295–324.!
[8]$Scholes, Robert. “Narration and Narrativity in Film.” Quarterly Review of Film Studies Vol.
1 (3), 1976. 283–96.!
[9]$Wolf, Werne. “Narrative and Narrativity: A Narratological Reconceptualization and its
Applicability to the Visual Arts.” Word & Image: A Journal of Verbal/Visual Enquiry, Vol.19
(3), 180–97.!
[10]$Ryan, Marie-Laure. “The Modes of Narrativity and Their Visual Metaphors.” Style Vol.26
(3), 1992. 368-87.!
[11]$Paterson, M. W. D. “Digital Scratch and Virtual Sniff.” In: The Smell Culture Reader. Ed.
Jim Drobnick. New York: Berg, 2006. 358–70.!
[12]$Gilbert, Avery. What the Nose Knows: The Science of Scent in Everyday Life. New York:
Crown, 2008. 147–69.!
[13]$Olofsson, Jonas K., Simon Niedenthal, Marie Ehrndal, Marta Zakrzewska, Andreas
Wartel, & Maria Larsson. “Beyond Smell-O-Vision: Possibilities for Smell-Based Digital
Media.” Simulation & Gaming Vol.48 (4), 2017. 455–79. !
[14 ]$Sherrington, Charles. S. The Integrative Action of the Nervous System. Yale University
Mrs. Hepsa Ely Silliman Memorial Lectures. New Haven: Yale UP, 1906. 226.!
[15]$Smith, Ewan St. John. “Advances in Understanding Nociception and Neuropathic Pain.”
Journal of Neurology Vol.265 (2), 2018. 231–38.!
[16]$Ledermann, Susan J. & Roberta L. Klatzky. “Haptic Perception: A Tutorial.” Attention,
Perception, & Psychophysics Vol.71 (7), 2009. 1439–59.!
[17]$Atwater, Riannon C. “Exteroception.” In: The Five Senses and Beyond. The Encyclopedia
of Perception. Ed. Jennifer L. Hellier. Santa Barbara: Greenwood, 2017. 152–55. 154–55.!
[18]$Tuthill, John C. & Eiman Azim. “Proprioception.” Current Biology Vol.28 (5), 2018. R194–
R203.!
[19]$Aukstakalnis, Steve. Practical Augmented Reality: A Guide to the Technologies,
Applications, and Human Factors for AR and VR. Boston: Addison-Wesley, 2017.!
[20]$Blade, Richard A. & Mary Lou Padgett. “Virtual Environments Standards and Ter-
minology.” In: Handbook of Virtual Environments: Design, Implementation, and Ap-
plications. Ed. Kelly S. Hale & Kay M. Stanney. Boca Raton: CRC, 20152. 23–35.!
[21]$Calleja, Gordon, Laura Herrewijn, & Karolien Poels. “Affective Involvement in Digital
Games.” In: Emotion in Games: Theory and Praxis. Ed. Kostas Karpouzis & Georgios N.
Yannakakis. Springer, 2016. 39–56.!
[22]$Swink, Steve. Game Feel: A Game Designer's Guide to Virtual Sensation. Burlington:
Morgan Kaufmann, 2004.!
[23]$Gomez, Patrick & Brigitta Danuser. “Relationships Between Musical Structure and
Psychophysiological Measures of Emotion.” Emotion Vol.7 (2), 2007. 377–87.!
[24]$Robinson, Jenefer. Deeper than Reason: Emotion and its Role in Literature, Music, and
Art. Oxford: Oxford UP, 2005. 310–13.!
[25]$Tarasti, Eero. Semiotics of Classical Music: How Mozart, Brahms, and Wagner Talk to Us.
Berlin: de Gruyter, 2012.!
[26]$Garrido, Sandra. Why Are We Attracted to Sad Music? London: Palgrave Macmillan,
2017.!
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/
! 8
J. Martin
[27]!Sharon E. Guttman, Lee A. Gilroy, & Randolph Blake. “Hearing What the Eyes See:
Auditory Encoding of Visual Temporal Sequences.” Psychological Science Vol.16 (3),
2005. 228–35."
[28] !Jacob Jolij & Maaike Meurs. “Music Alters Visual Perception.” PLoS ONE Vol.6 (4):
e18861, 2011. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.
0018861"
[29]!Zadra, Jonathan R. & Gerald Clore. “Emotion and Perception: The Role of Affective
Information.” Wiley Interdisciplinary Reviews. Cognitive Science Vol.2 (6). 676–85."
[30]!Kraus, Michael W. “Voice-Only Communication Enhances Empathic Accuracy.”
American Psychologist Vol.72 (7), 2017. 644–54."
[31]!Rubin, David C. & Jennifer M. Talarico. “A Comparison of Dimensional Models of
Emotion". Memory Vol.17 (8), 2009. 802–08."
[32]!Mobbs, Dean, Nikolaus Weiskopf, Hakwan C. Lau, Eric Featherstone, Ray J. Dolan, &
Chris D. Frith. “The Kuleshov Effect: The Influence of Contextual Framing on Emotional
Attributions.” Social Cognitive and Affective Neuroscience Vol.1 (2), 2006. 95–106."
[33]!Barratt, Daniel, Anna Cabak Rédei, Åse Innes-Ker, & Joost van de Weijer. “Does the
Kuleshov Effect Really Exist? Revisiting a Classic Film Experiment on Facial Expressions
and Emotional Contexts.” Perception Vol.45 (8), 2016. 847–74."
[34]!Baranowski, Andreas M., & Heiko Hecht. “The Auditory Kuleshov Effect: Multisensory
Integration in Movie Editing.” Perception Vol 46 (5), 2017. 624–31."
[35]!Dutton, Donald G. & Arthur P. Aron. “Some Evidence for Heightened Sexual Attraction
Under Conditions of High Anxiety.” Journal of Personality and Social Psychology 30 (4),
1974. 510–17."
[36]!Schachter, Stanley & Jerome Singer. “Cognitive, Social, and Physiological Determinants
of Emotional State". Psychological Review Vol.69 (5), 1962. 379–99."
[37]!Cotton, John. “A Review of Research on Schachter’s Theory of Emotion and the
Misattribution of Arousal.” European Journal of Social Psychology Vol.11 (4), 1981. 365–
97."
[38]!Savitsky, Kenneth, Victoria Husted Medvec, Ann E. Charlton, & Thomas Gilovich. “‘What,
Me Worry?’: Arousal, Misattribution, and the Effect of Temporal Distance on
Confidence.” Personality and Social Psychology Bulletin Vol.24 (5), 1998. 529–36."
[39]!White, Gregory L., Sanford Fishbein, & Jeffrey Rutsein. “Passionate Love and the
Misattribution of Arousal.” Social Psychology Vol.41 (1), 1981. 56–62."
[40]!Allen, James B., Douglas T. Kenrick, Darwyn E. Linder, Michael A. McCall. “Arousal and
Attraction: A Response-Facilitation Alternative to Misattribution and Negative-Rein-
forcement Models.” Journal of Personality and Social Psychology Vol.57 (2), 1989. 261–
70."
[41]!Marin, Manuela M., Raphaela Schober, Bruno Gingras, & Helmut Leder. “Misattribution
of Musical Arousal Increases Sexual Attraction Towards Opposite-Sex Faces in
Females.” PLoS One Vol.12 (9): e0183531, 2017. https://www.ncbi.nlm.nih.gov/pmc/arti-
cles/PMC5593195/ "
Electronic preprint July 2016 | https://betweendrafts.com/justdrafts/2016/07/08/academic-paper-making-sense-
juxtaposing-visual-auditory-and-kinesthetic/