Shitsukan — the Multisensory Perception of Quality

Abstract

We often estimate, or perceive, the quality of materials, surfaces, and objects, what the Japanese refer to as ‘shitsukan’, by means of several of our senses. The majority of the literature on shitsukan perception has, though, tended to focus on the unimodal visual evaluation of stimulus properties. In part, this presumably reflects the widespread hegemony of the visual in the modern era and, in part, is a result of the growing interest, not to mention the impressive advances, in digital rendering amongst the computer graphics community. Nevertheless, regardless of such an oculocentric bias in so much of the empirical literature, it is important to note that several other senses often do contribute to the impression of the material quality of surfaces, materials, and objects as experienced in the real world, rather than just in virtual reality. Understanding the multisensory contributions to the perception of material quality, especially when combined with computational and neural data, is likely to have implications for a number of fields of basic research as well as being applicable to emerging domains such as, for example, multisensory augmented retail, not to mention multisensory packaging design.

Full text

Multisensory Research 33 (2020) 737–775 brill.com/msr
Review
Shitsukan — the Multisensory Perception of Quality
Charles Spence ∗
Department of Experimental Psychology, Anna Watts Building, University of Oxford, Oxford,
OX2 6GG, UK
Received 17 December 2019; accepted 29 January 2020
Abstract
We often estimate, or perceive, the quality of materials, surfaces, and objects, what the Japanese
refer to as ‘shitsukan’, by means of several of our senses. The majority of the literature on shitsukan
perception has, though, tended to focus on the unimodal visual evaluation of stimulus properties. In
part, this presumably reflects the widespread hegemony of the visual in the modern era and, in part, is
a result of the growing interest, not to mention the impressive advances, in digital rendering amongst
the computer graphics community. Nevertheless, regardless of such an oculocentric bias in so much
of the empirical literature, it is important to note that several other senses often do contribute to the
impression of the material quality of surfaces, materials, and objects as experienced in the real world,
rather than just in virtual reality. Understanding the multisensory contributions to the perception of
material quality, especially when combined with computational and neural data, is likely to have
implications for a number of fields of basic research as well as being applicable to emerging domains
such as, for example, multisensory augmented retail, not to mention multisensory packaging design.
Keywords
Shitsukan, quality perception, material perception, multisensory, crossmodal, incongruency
1. Introduction: Visual Hegemony in the Study of Material Perception
Komatsu and Goda (2018) published a review of the literature on ‘Shitsukan’.
This Japanese term translates roughly as ‘a sense of material quality’ or ‘mate-
rial perception’. Importantly, however, while the abstract to their paper stresses
the important contribution that the non-visual senses make to shitsukan per-
ception, Komatsu and Goda’s review primarily deals just with unisensory
visual contributions to its evaluation. In particular, these researchers focus on
*E-mail: charles.spence@psy.ox.ac.uk
©Spence, 2020 DOI:10.1163/22134808-bja10003
This is an open access article distributed under the terms of the CC BY 4.0 license.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

738 C. Spence / Multisensory Research 33 (2020) 737–775
both behavioural data and the recently emerging body of neuroimaging lit-
erature that has attempted to determine the neural substrates responsible for
the perception of various aspects of material quality, such as visual glossiness
(Hunter, 1975; see Chadwick and Kentridge, 2015, for a review), translucency
(Chadwick et al., 2017), roughness (Ho et al., 2008), and/or iridescence (Sha-
ran et al., 2014; see Anderson, 2011, and Fleming, 2014, for reviews). There
is also interest in the visual perception of different classes of materials such as
woods, plastics, pearls, etc. (e.g., Tani et al., 2014).
The majority (but by no means all) of the references cited by Komatsu and
Goda (2018) deal only with unimodal studies of visual shitsukan at either the
behavioural/psychophysical and/or neural levels. While such a visual bias is,
of course, widespread across many areas of science (see Hutmacher, 2019, for
a recent review; and Fraser, 1892, for a much earlier commentary), the very
ubiquity of this visual hegemony (see Levin, 1993; Mirzoeff, 1999) certainly
does not mean that it should go unchallenged. This visual bias in shitsukan
research is presumably driven, at least in part, by the growing interest, not
to mention the impressive advances in computer graphics and the visual ren-
dering of complex material properties such as glossiness and iridescence. As
such, Komatsu and Goda’s (2018) review would implicitly seem to be targeted
at the computer graphics and computational vision communities. By contrast,
the present review is directed more toward those working at the interface of
augmented retail and other more product-related applications, such as, for ex-
ample, those working on the development of multisensory product packaging.
While there has undoubtedly been some progress in the field of haptic ren-
dering (e.g., see Lin and Otaduy’s, 2008, volume on this theme; see also Bicchi
et al., 2008; Salisbury et al., 1995) (see Note 1), the field of haptic or, for that
matter, auditory (see Aramaki et al., 2011; Klatzky et al., 2000; Serafin et al.,
2011) digital rendering has not advanced anything like as much as might have
been hoped, even over the last decade (e.g., Jones and Ho, 2008; see Gallace
and Spence, 2014, or Parisi, 2018, for reviews of the tactile/haptic domain).
There are a number of reasons for this asymmetry. In part, it presumably re-
lates to limitations in the available technology for tactile/haptic stimulation.
However, it is also worth noting that the estimated bandwidths of the visual
and tactile modalities are radically different (see Table 1 for a summary). What
is also unique about haptic rendering is that it is bidirectional and, as such, the
bandwidth of the feedback loop is critical to the fidelity of what is rendered
(e.g., Salisbury et al., 2004). One might also consider here how much com-
mercial interests, and their associated investments, play into this space. That
said, though, it is worth noting that touch-screen technology has rapidly been
incorporated into everyday devices in recent years (at least three billion touch-
enabled devices worldwide by 2015 according to Immersion Technology; see
https://www.immersion.com/3-billion-devices-have-touch/).
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 739
Table 1.
Table summarizing the number of sensors, number of afferents, information transmission
rates/channel capacity (from Zimmerman, 1989), the putative percentage of attentional capture
(from Heilig, 1992) and the percentage of neocortex (Felleman and Van Essen, 1991) relative
to each sensory modality (though see Hsiao, 1998, for similarities between vision and touch).
(Reprinted from Gallace et al., 2012.)
Sensory
system
No. of
sensors
No. of
afferents
Channel
capacity
(bits/s)
Psychophysical
channel capacity
(bits/s)
%
Attentional
capture
%
Neocortex
Vision 2*1082*10610740 70% 55%
Audition 3*1042*10410530 20% 3.4%
Touch 1071061065 4% 11.5%
Taste 3*1071031031(?) 1% 0.5%
Smell 7*1071051051(?) 5% n.a.
There are currently a number of intriguing possibilities as far as augmented
retail applications are concerned, where stimulating more than just the con-
sumer’s eyes holds the promise of increasingly engaging multisensory applica-
tions (e.g., see Heller et al., 2019; Leswing, 2016; Overmars and Poels, 2015;
Xiao et al., 2016; see Petit et al., 2019, for a review). To be absolutely clear,
the primary audience for this particular review is those working (or interested)
in applied multisensory domains, including those considering how best to ren-
der material perception, as well as the rapidly-emerging field of multisensory
packaging design (see Velasco and Spence, 2019, for reviews). Neuroimaging
studies of multisensory shitsukan will not be covered in any detail here, in
part because there is little that specifically relates to multisensory shitsukan
perception, and in part because the unisensory, primarily visual, literature has
been summarized so thoroughly by Komatsu and Goda (2018).
2. Non-Visual Contributions to Shitsukan
It can be argued that what is missing from Komatsu and Goda’s (2018) other-
wise excellent review is a full discussion of the role played by the non-visual
senses in the perception of material quality (cf. Fujisaki, 2020). At best, what
one gets is an acknowledgement that certain material properties can only be
assessed, or at least are much easier to assess, via another sense, such as touch
(be it passive or active, the latter, note, commonly if not universally referred
to as haptics). In particular, attributes such as weight, compressability, tem-
perature, and fine (or microgeometric) surface texture are typically easier to
ascertain reliably via contact with the skin surface and/or haptic exploration
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

740 C. Spence / Multisensory Research 33 (2020) 737–775
rather than solely by means of visual inspection (e.g., Drucker, 1988; Gal-
lace and Spence, 2014; Guest and Spence, 2003a; Krishna and Morrin, 2008).
There has also long been interest in the tactile/haptic discrimination of differ-
ent materials, and more specifically different qualities of, for example, wool
(Binns, 1926) or wood flooring (Berger et al., 2006).
These material properties are often distinguished from an object’s geomet-
ric properties (shape, curvature, orientation, size and volume) the perception
of which is generally based on visual cues (Kahrimanovic et al., 2010). Both
vision and touch can be used to determine shape (Lacey and Sathian, 2014;
Norman et al., 2004). According to the neuroimaging research, there would
appear to be parallel pathways in the brain for the processing of surface prop-
erties and the form of objects (Cant and Goodale, 2007; Sathian et al., 2011).
In their review, Komatsu and Goda highlight the growing body of evidence
suggesting that certain material qualities may be represented in the same brain
regions, regardless of the sensory modality through which those qualities hap-
pen to be perceived (e.g., Eck et al., 2013; Goda et al., 2016; see also Whitsel
et al., 1989). Similarly, the brain areas involved in determining various object
properties, or a person’s evaluation of those properties (e.g., in terms of aes-
thetic appreciation), may well turn out to be shared between the senses (e.g.,
Brown et al., 2011; see also Schifferstein and Hekkert, 2011). A similar claim
can also be made with regards to sensory attributes such as surface texture and
form (Goda et al., 2016; Pérez-Bellido et al., 2018; Podrebarac et al., 2014;
see also Eck et al., 2013; Sun et al., 2016; Yau, Hollins and Bensmaia, 2009).
3. Crossmodal Influences on Judgements of Visual Quality
Beyond conveying information concerning those physical attributes of a stim-
ulus that cannot be ascertained visually, it is important to note that even
what we consider to be visual judgements, or rather judgements of visually-
determined material properties, are often influenced by whatever other sensory
inputs are present at around the same time, no matter whether we realize it or
not (e.g., Adams et al., 2016; Hagtvedt and Brasel, 2016); and mostly the
evidence suggests that we do not (e.g., Laird, 1932; Li et al., 2007). So, for
example, it is well known amongst laundry detergent manufacturers (such as
Unilever) that delivering ‘brilliant whites’ is about so much more than simply
what the customer sees. Perhaps counterintuitively, adding the right ‘clean’
fragrance really can help to make one’s whites look brighter (see Vickers and
Spence, 2007). In fact, there is now a rich body of empirical data demonstrat-
ing the modulatory role of scent in our evaluation of the physical attributes
(e.g., age, attractiveness, and gender) of other people (e.g., Demattè et al.,
2007a; Li et al., 2007; McGlone et al., 2013).
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 741
Adding the right scent has also been shown to influence people’s ratings of
fabric softness and material quality (e.g., Churchill et al., 2009; Demattè et al.,
2006; Laird, 1932). In recent years, the presence of congruent versus incon-
gruent scents has also been found to affect people’s perception of a variety of
material properties as well as their aesthetic response to those materials (e.g.,
Bone and Jantrania, 1992; Bosmans, 2006; Demattè et al., 2007b; Krishna
et al., 2010; Zellner et al., 2008; though see also Schifferstein and Michaut,
2002).
We typically inspect objects and materials visually first. Furthermore, such
visual impressions likely anchor the subsequent haptic/multisensory evalua-
tion (e.g., Buckingham et al., 2009; Wijntjes et al., 2019; Xiao et al., 2016;
see Atlas and Wager, 2013, and Piqueras-Fiszman and Spence, 2015, for re-
views). At the same time, however, it is also important to recognize that this
does not mean that material quality cannot be assessed, or inferred, by the
cues available to the other senses too (e.g., Baumgartner et al., 2013; De-
cré and Cloonan, 2019; Fujisaki et al., 2014; Picard, 2007; Xiao et al., 2016;
Yanagisawa and Takatsuji, 2015; Yanagisawa and Yuki, 2011).
As Komatsu and Goda (2018, p. 330) note: “Every sensory modality is
involved in material perception.Not only that,material perception also has
crossmodal aspects.For e xam pl e,when we see a sweater made of fine wool,
we can perceive that it will be soft and warm,or we can sense that a metal
cup will be cold and hard to the touch.” They go on to suggest that: “Some
material properties,such as microscale roughness,hardness,coldness,and
weight,are nonvisual and cannot be directly sensed visually.Nevertheless,
interestingly,humans can accurately estimate such nonvisual properties from
the visual appearance of materials,in a way that correlates with those hapti-
cally estimated through touching them (Baumgartner et al., 2013).” (Komatsu
and Goda, 2018, p. 340; see also Yanagisawa and Takatsuji, 2015).
A growing body of empirical research now shows that the impressions of-
fered by the non-visual senses sometimes also contribute to, or modify, the
‘visual’ impression of various material qualities (e.g., Hagtvedt and Brasel,
2016; Jansson-Boyd and Marlow, 2007; Laird, 1932; Murakoshi et al., 2013).
Oftentimes, though, people do not seem to realize just how much informa-
tion may be carried by the non-visual (and hence often unattended, or less
attended) senses.
Interest in the multisensory assessment of material qualities stretches back
to the early days of experimental psychology, as documented in the seminal
work of English scientist Henry Binns (e.g., Binns, 1934, 1937). The latter
was charged with assessing the manner in which experts assessed the quality
of woollen tops in the mills, a matter at the time that was of great practical (not
to mention pecuniary) interest to the mill owners. More than 80 years ago,
Binns’ research had already demonstrated the importance of both the visual
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

742 C. Spence / Multisensory Research 33 (2020) 737–775
and tactile assessment of quality in expert assessors. In one intriguing recent
study on a related theme, Xiao et al. (2016) had their participants match pho-
tographs of fabric samples to the feel of physical fabric samples. Removing
colour was found to reduce accuracy, especially when the images contained
3-D folds. Overall, images of draped fabrics, which revealed 3-D shape in-
formation, resulted in better matching accuracy than those images displaying
flattened fabrics instead. Overall, therefore, it would appear that people use
chromatic gradients to infer tactile fabric properties, at least if they happen to
be available.
4. Sensory Dominance and Computational Accounts of Multisensory
Integration
Importantly, however, the perception of material quality, established on the ba-
sis of non-visual cues, may well be overridden by what we see (Fujisaki et al.,
2014) — a phenomenon known as ‘visual dominance’ (Posner et al., 1976).
So, for example, according to influential early research, our impression of the
size and shape of an object may be completely dominated by vision, often
overriding any discrepant tactile/haptic cues that happen to be present (Rock
and Harris, 1967; Rock and Victor, 1964; see Spence, 2011a, for a review).
Ernst and Banks (2002) brought some much-needed mathematical rigour to
the field of sensory (i.e., visual) dominance research by demonstrating that
maximum-likelihood estimation provides an excellent quantitative account of
the integration of visual and tactile/haptic cues in relation to size/length judg-
ments. In their study, participants had to judge the height of a bar that could
be seen and also felt between the thumb and index finger of one hand (deliv-
ered virtually by means of two force-feedback devices). Adding noise to the
visual signal resulted in the participants increasingly relying on haptic infor-
mation when making their judgements. According to the maximum-likelihood
estimation account of sensory dominance, the human brain combines sensory
inputs in a manner that is very close to that of a statistically optimal multisen-
sory integrator. That is, the multisensory integration of disparate unisensory
inputs appears to maximally reduce the uncertainty of, or variance associated
with, our multisensory estimates of external stimulus qualities (given that all
sensory estimates are intrinsically noisy). In fact, the maximum-likelihood ac-
count provides a surprisingly good account of the relative contribution of each
of the senses to multisensory perception in a variety of different settings, and
for a variety of different combinations of stimulus modalities (e.g., Alais and
Burr, 2004). Nevertheless, there may still be some residual role for directed
attention in explaining the patterns of sensory dominance that are sometimes
observed (Meijer et al., 2019).
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 743
However, one problem that was not tackled by Ernst and Banks’ (2002)
seminal work concerns the binding problem, namely the problem of which
cues should be integrated, and which should be kept separate. According to
subsequent research on causal inference, the unity/segregation decision can be
resolved probabilistically by means of the incorporation of a variety of priors
(see Körding et al., 2007; though see also Chen and Spence, 2017). There
has also been research into the multisensory (especially visual–tactile/hap-
tic) integration of surface roughness (often using sandpaper samples). Indeed,
over the years, an extensive body of research has investigated visual–tactile
interactions in the perception of roughness (e.g., Guest and Spence, 2003b;
Heller, 1982; Jones and O’Neil, 1985; Lederman and Abbott, 1981; Lederman
and Klatzky, 2004; Lederman, Thorne and Jones, 1986; Warren and Rossano,
1991; Werner and von Schiller, 1932). Intriguingly, based on the available em-
pirical evidence, it would appear that while tactile cues tend to dominate when
the senses are put into conflict as far as microgeometric surface textures are
concerned, vision typically dominates for the perception of macrogeometric
surface properties (see also Klatzky et al., 1993). This somewhat unusual pat-
tern of dominance presumably reflects the relative precision of the two senses
at different scales (Klatzky and Lederman, 2010). Lederman et al. (1986)
proposed a weighted averaging model of visuotactile texture perception. The
Bayesian account has now been extended to the perception of surface tex-
ture (e.g., Yanagisawa and Takatsuji, 2015). However, given that the Bayesian
causal inference account of shape/texture perception has been reviewed exten-
sively elsewhere, that will not be covered in any more detail here.
4.1. Audiovisual Contributions to Multisensory Shitsukan Perception:
Computational Account
While much of the research documenting multisensory contributions to ma-
terial perception has focused on either visuotactile or audiotactile integration,
audiovisual integration is also of interest. It is also perhaps the easiest modality
combination to render digitally at the present time (see Fujisaki et al., 2014;
see also Etzi et al., 2018; Fenko et al., 2011; Gerdes et al., 2014). In one fasci-
nating study, Fujisaki et al. had 16 participants rate auditory (impact sounds),
visual and audiovisual stimuli depicting a variety of impact events. All pos-
sible combinations of six visible materials were crossed with eight different
possible impact sounds. The researchers wanted to know which sense would
dominate in terms of people’s perception of the material category when view-
ing one material combined with the impact sound of another (incongruent)
material. The participants had to rate how likely it was that the experimen-
tal stimuli that they were presented with were to indicate one of 13 material
categories. Participants rated each of the six visual stimuli and each of the
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

744 C. Spence / Multisensory Research 33 (2020) 737–775
eight impact sounds when presented in isolation. They also had to rate all 48
possible audiovisual combinations of the two unisensory stimuli.
The results of this study indicated strong interactions between the senses in
terms of participants’ material perception. So, for example, when the appear-
ance of glass was paired with an impact sound from a bell pepper, the resulting
audiovisual stimulus was rated as seeming like transparent plastic (see Fig. 1).
According to Fujisaki et al. (2014, p. 1): “Rating material-category likeli-
hoods follow a multiplicative integration rule in that the categories judged
to be likely are consistent with both visual and auditory stimuli.On the other
hand,rating-material properties,such as roughness and hardness,follow a
weighted average rule.Despite a difference in their integration calculations,
both rules can be interpreted as optimal Bayesian integration of independent
audiovisual estimations for the two types of material judgment,respectively.”
Here, though, it is perhaps worth noting that research from elsewhere in
the field of multisensory perception has shown that the result when presenting
incongruent pairs of multisensory stimuli may itself depend on the context in
which they happen to be presented. In particular, Gau and Noppeney (2016)
documented a reduced McGurk effect when audiovisual McGurk stimulus
pairs were embedded in a context of incongruent audiovisual speech stimuli
compared to when they were embedded in a stream of congruent speech stim-
uli instead. It may therefore be relevant to note that in Fujisaki et al.’s (2014)
study, the vast majority of the multisensory material pairs that were presented
to participants were, in some sense, incongruent (i.e., originating from pairs
of stimuli that did not belong together) meaning perhaps that less crossmodal
interaction will be observed.
4.2. Crossmodal Correspondences and Multisensory Shitsukan Perception
Consistent with the view that many non-visual associations may be stored as
learned associations, or in some cases crossmodal correspondences (see also
Fenko et al., 2010b; Spence, 2011b), we internalize the statistics of the en-
vironment (Parise et al., 2014; Peeva et al., 2004; Peters et al., 2015, 2018).
Hence, crossmodal relations between material properties will likely be rapidly
internalized as coupling priors (Chen and Spence, 2017; Ernst, 2007; Komatsu
and Goda, 2018; Spence, 2011b; Yuan et al., 2017).
It should, though, be borne in mind that such crossmodal relations may re-
late to putatively amodal material properties, such as surface texture or form,
or else to what Walker-Andrews (1994) refers to as arbitrary crossmodal re-
lations, such as the ring tone that is associated with your mobile phone, say,
or the lemon or pine fragrance added to many cleaning products. There is
also an interesting, and perhaps orthogonal question here as to whether the
associations (or correspondences) between the senses are all learnt from the
statistics of the environment (or the marketplace; e.g., see Ye et al., 2019), and
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 745
Figure 1. Examples of the obtained profiles in the material category experiment (average of
16 participants). (a) An auditory-induced visual material category–perception change. Differ-
ent material categories were perceived for the same visual stimulus with different sounds.
(b) A vision-induced auditory material category–perception change. Different material cate-
gories were perceived for the same auditory stimulus with different visual stimuli. (Figure and
legend reprinted from Fujisaki et al., 2014, Fig. 4.)
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

746 C. Spence / Multisensory Research 33 (2020) 737–775
hence might, in some sense, be considered arbitrary, like the association be-
tween packaging attributes and likely product qualities, or push-button sounds
and/or whether any have much older precedents in human development (and
hence which might be considered by some as putatively innate; see LaBonte,
2009; Meert et al., 2014; Saad and Gill, 2000; Spence, 2011b). One might also
want to draw a distinction here between those cases where the different senses
are picking up on the same putatively amodal stimulus property (such as visual
and tactile/haptic estimates of shape, size, or surface texture), versus on non-
redundant stimulus dimensions (as in the case of crossmodally corresponding
stimulus dimensions, such as, for example, the corresponding dimensions of
auditory pitch and visual size, or elevation; see Deroy et al., 2018; see also
Walker et al., 2010, 2017, in press).
The sections that follow selectively review some of the most intriguing
evidence concerning the role played by the non-visual senses in shitsukan
(Note 2).
5. Unisensory Quality Assessments Beyond the Visual Modality
5.1. Tactile Shitsukan Perception
Over the course of the last century, many researchers have investigated the
perception of shitsukan (though the research is often not described as such,
outside Japan) by means of unisensory cues presented in one of the non-visual
senses. As a matter of fact, the majority of this research has focused on the
tactile, or haptic, perception of material quality (e.g., Binns, 1926; Culbert
and Stellwagen, 1963; Hollins et al., 1993; Ludden and Van Rompay, 2015;
Okamoto et al., 2013; Philippe et al., 2004; Picard et al., 2003; Yoshida,
1968a, b, c). This is what we might call the feel the quality. Much of the
unisensory tactile/haptic research has focused on an assessment of the ma-
terial perception of the quality of fabrics. In terms of the affective response
to materials, it turns out that softness is important, especially for those gar-
ments worn close to the skin (e.g., Chang et al., 2015; Kergoat et al., 2012;
see also Etzi et al., 2014; Teli, 2015). However, there is also a large body of
research concerning the tactile/haptic aspects of object recognition/perception
(e.g., Karlsson and Velasco, 2007; Klatzky et al., 1985; Lederman, 1982; Son-
neveld and Schifferstein, 2008; Spence and Gallace, 2011). There has even
been some limited research in the field of tactile aesthetics that may be rele-
vant here (Gallace and Spence, 2011; see also Lindauer, 1986; Lindauer et al.,
1986).
Key material dimensions in the tactile/haptic modality include surface
roughness, surface texture, compliance, weight and temperature (or thermal
diffusivity, Bergmann Tiest and Kappers, 2009; see also Bergmann Tiest and
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 747
Kappers, 2006; Bhatta et al., 2017; Chen et al., 2009; Goebl et al., 2014)
and they are fundamental to haptic perception of object properties (Bergmann
Tiest, 2010; see also Fujisaki, 2020). Beyond that, there are also the primar-
ily tactile/haptic properties of liquid materials, namely viscosity and wetness
(Bergmann Tiest, 2015). One of the other important judgements that a person
can make as far as the material properties of a surface or object is concerned
relates to its perceived ‘naturalness’. Over the last decade or so, a number of
researchers have investigated the relative contribution of different senses to the
perception of this material quality (e.g., Binninger, 2017; Labbe et al., 2013;
Nikolaidou, 2011; Overvliet and Soto-Faraco, 2011; Overvliet et al., 2016;
Whitaker et al., 2008; see also Fujisaki et al., 2015; Kanaya et al., 2016). Here
it should be noted that there is a growing interest in how other packaging cues,
such as a matte finish, may also be linked to, and hence help to convey product
naturalness (e.g., Han, 2018; Marckhgott and Kamleitner, 2019). Understand-
ably, many companies have also been interested in the question of whether it
is possible to render/manufacture a natural finish (e.g., to product packaging
or building materials).
As far as tactile perception is concerned, the impression of shitsukan may
depend on the particular region of the skin surface that is used to evaluate
a material (see Ackerley et al., 2012; Etzi et al., 2016). Not only are differ-
ent sensitivities documented at different skin sites, but some tactile receptors,
namely C-tactile afferents, are only found in the hairy skin (e.g., see Löken
et al., 2009; McGlone and Spence, 2010). Several researchers have demon-
strated that one and the same material may be rated quite differently as a
function of the skin surface against which it makes contact (Ackerley et al.,
2014; Essick et al., 2010; Etzi et al., 2016). At the same time, however, some
surprisingly robust individual differences in the ‘Need for Touch’ (NFT) have
also been identified by researchers (e.g., Peck and Childers, 2003a, b). Peck
and Childers (2003a, p. 431) define it as “a preference for the extraction and
utilization of information obtained through the haptic system”. An individual’s
NFT is typically assessed by means of their response to a standardized series
of statements. Those scoring higher (i.e., agreeing more with the questions)
are rated as higher in their autotelic need for touch. The sorts of statements
used by Peck and Childers to pick out those high in the autotelic NFT include
the following: ‘Touching products can be fun’; and ‘I find myself touching all
kinds of products in stores’. According to Peck and Childers (2008, p. 207):
“The instrumental dimension of NFT refers to those aspects of touch that re-
flect outcome directed touch with a salient purchase goal... Autotelic touch
involves a consumer seeking fun,sensory stimulation,and enjoyment with no
purchase goal necessarily salient”.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

748 C. Spence / Multisensory Research 33 (2020) 737–775
In the years since the NFT framework was put forward, a number of studies
have documented that it provides a useful means of distinguishing meaning-
fully between different groups of consumers (Krishna and Morrin, 2008; see
also Citrin et al., 2003). In practice, differences in the NFT typically mean
that people are differentially affected by, and hence seek out, the tactile/haptic
qualities of an object or material (Ackerman, 2016; Childers and Peck, 2010;
Spence, 2019; Workman, 2010). Although it remains unclear, the suggestion
is that these individual differences in the NFT presumably reflect more cogni-
tive (i.e., central, rather than peripheral receptor-based) individual differences.
Intriguingly, similar individual differences have not been reported for the other
higher ‘spatial’ senses.
5.2. Auditory Shitsukan Perception
A separate body of experimental research has assessed auditory shitsukan per-
ception (e.g., Björk, 1985; Giordano and McAdams, 2006; McDermott and
Simoncelli, 2011; Zhang et al., 2017). Auditory cues turn out to play an impor-
tant role in the assessment of product quality (Björk, 1985), what one might
be minded to describe as the sound of quality. In terms of the category of
material, Giordano and McAdams assessed people’s ability to identify ob-
ject materials on the basis of impact sounds (that is, when something strikes
an object). These researchers demonstrated that people could discriminate al-
most perfectly between the various material categories on the basis of sound
(e.g., steel–glass versus wood–plexiglass). The available functional magnetic
resonance imaging (fMRI) research here suggests that a sub-region in the
ventro-medial pathway appears to be specialized for the task of auditory ma-
terial perception (Arnott et al., 2008).
A particularly rich vein of research on auditory assessment of material prop-
erties relates specifically to the material qualities of foods that are associated
with the sounds resulting from oral mastication (see Spence, 2015; Zampini
and Spence, 2004, for a review); think here only of the qualities of crisp,
crunchy, crackly, squeaky that are sometimes experienced in food. Indeed, as
Fujisaki et al. (2014, p. 1) have noted: “material perception is a critical abil-
ity for animals to properly regulate behavioural interactions with surrounding
objects (e.g., eating)” (see also Nagano et al., 2014).
In terms of other material properties, research has been conducted to show
that while the majority of people do not believe that they could determine
whether a liquid is hot or cold simply by listening to the sounds of pouring (cf.
Stuckey, 2012), most of us turn out to be significantly better than chance at this
task. Under forced-choice conditions, using nothing more than auditory cues,
people are able to correctly distinguish the sound of recently boiled water from
water that has been removed from the fridge instead (Velasco et al., 2013a, b;
see also Wildes and Richards, 1988, on the recovery of material properties
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 749
from sound). In the case of liquids such as water, viscosity changes due to
temperature lead to a distinctive change in pitch (Parthasarathy and Chhapgar,
1955). It would appear, then, that we internalize this, and other, environmental
statistics (Peeva et al., 2004), despite the fact that we typically tend to use
thermal and/or visual cues to make such an assessment of the temperature of
an environmental stimulus (Note 3) (cf. Fenko et al., 2010a; Wastiels et al.,
2012).
The sounds of opening and closing product packaging also convey useful
information (see Wang and Spence, 2019, for a review). Recently, for example,
it has been demonstrated that people tend to rate wine as tasting better (i.e.,
as being of higher quality) when they hear the sound of a cork-stoppered bot-
tle being opened rather than when they hear a screw-cap bottle being opened
instead (Spence and Wang, 2017). In fact, there is a rich literature of research
that has investigated the influence of the sounds of packaging opening and
usage in the food and beverage category (Spence and Wang, 2015).
The sound of car engines, not to mention car doors, and even the sound of
the dashboard when tapped with the knuckles, are important and longstanding
areas of psychoacoustics research too (Kanie et al., 1987; Montignies et al.,
2010; see Spence and Zampini, 2006, for a review). There is also an intriguing
literature on the design of push-button sounds too (Altinsoy, 2020). Academic
researchers have worked hard to modify the sounds made by products such
as cigarette lighters (Lageat et al., 2003), air-conditioning units (Susini et al.,
2004), and vacuum cleaners (Wolkomir, 1996). Even the sound of closure of
the mascara, or the distinctive pop of the Snapple bottle has been engineered to
sound ‘just so’ (Byron, 2012), to provide, in other words, the sound of quality
(e.g., Ozcan and van Egmond, 2012; see also Avanzini and Crosato, 2006;
Kim et al., 2007).
5.3. Olfactory Shitsukan Perception
As far as the chemical senses are concerned, researchers have investigated
the perception of the olfactory quality of scents/perfumes (e.g., studying the
perception of complexity, and aesthetic appreciation; e.g., Rabin, 1988; Schiff-
mann, 1974). There is also a separate literature on those factors influencing the
assessment of the (material) quality of food and drink. The latter, note, often
being subsumed within the literature on sensory science/sensory studies, and
hence published in journals such as Food Quality & Preference and the Jour-
nal of Sensory Studies.
5.4. Interim Summary
As the results reviewed in this section demonstrate, shitsukan perception oc-
curs in the tactile, auditory, and olfactory modalities when experienced uni-
modally (see also Haase and Wiedmann, 2018). Of course, that said, the
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

750 C. Spence / Multisensory Research 33 (2020) 737–775
‘million dollar question’ here is whether those assessments of shitsukan, or
material quality, that are made under unisensory conditions have any predic-
tive value as far as what people will perceive, or report, under conditions where
multiple senses may be used in product evaluation (Ballesteros et al., 2005).
Indeed, given that visual dominance is such a ubiquitous feature of our object
identification (Posner et al., 1976; Spence, 2011a), the contribution of non-
visual cues to shitsukan perception might well be expected to be less important
than might perhaps be suggested on the basis of the unisensory studies that
have been reviewed above (though see also Hershberger and Misceo, 1996;
Komatsuzaki et al., 2016).
6. Multisensory Contributions to Shitsukan Perception
While the unisensory assessment of shitsukan undoubtedly constitutes an in-
teresting line of laboratory research, out in the real world, our experience
and evaluation of material properties is typically based on multisensory cues.
Importantly, a large body of empirical research demonstrates that any product-
intrinsic (not to mention some product-extrinsic) sensory inputs are com-
bined to deliver multisensory shitsukan perception (see also Qiao et al., 2014;
Schütte et al., 2008). Understandably, many researchers have been interested
in trying to understand the rules of multisensory integration and how they re-
late to the perception of shitsukan/material quality (e.g., Fujisaki et al., 2014;
Lederman et al., 1986; cf. Ernst and Banks, 2002). As was mentioned earlier,
Bayesian causal inference, built on the Maximum-Likelihood Estimation ap-
proach (Ernst and Banks, 2002), has seemingly done an excellent job in this
regard (e.g., Fujisaki et al., 2014; Peters et al., 2018; Yanagisawa and Takat-
suji, 2015).
However, while vision normally dominates, it is important to note that vi-
sual cues do not always allow for the accurate prediction of non-visual material
properties. As Fujisaki et al. (2014) suggest, vision is more useful for as-
sessing surface properties whereas auditory cues may be more informative
concerning the internal properties of a material or object. This was shown by
Wastiels et al. (2013) in those participants (studying architecture) in the case
of the estimated properties of various building materials. Indeed, while visual
cues tend to dominate multisensory material perception, there are also some
notable exceptions. So, for instance, Adams et al. (2016) have reported that
visual glossiness can be affected by the haptic slipperiness of a surface.
Nagai, Matsushima, Koida, Tani, Kitazaki and Nakauchi (2015) have re-
ported that it takes longer for people to process non-visual than visual material
properties, though this presumably has a lot to do with differences in the man-
ner of exploration in the different senses (Owens et al., 2016; Sun et al., 2016).
Being able to ascertain the felt weight of an object or especially product in the
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 751
hand is undoubtedly important in terms of shitsukan, given that it turns out to
be one of the key factors influencing the perception of quality (e.g., Jostmann
et al., 2009; Kampfer et al., 2017; Michel et al., 2015; Piqueras-Fiszman and
Spence, 2012; Schneider et al., 2011; and see Spence and Piqueras-Fiszman,
2011, for a review).
6.1. Auditory Contributions to Multisensory Shitsukan Perception
In terms of auditory contributions to multisensory shitsukan, the perception of
felt texture is often influenced by modifying any sounds that are elicited by the
interaction (e.g., Altinsoy, 2008, 2020; Guest et al., 2002; Jousmäki and Hari,
1998; Suzuki et al., 2006, 2008; and again see Werner and von Schiller, 1932,
for early work in this area). For instance, several studies of the ‘parchment
skin’ illusion have revealed that people’s perception of their own skin can be
modified simply by changing the sound of the interaction when they rub their
hands together in front of a microphone (see also Senna et al., 2014, on the
marble hand illusion). Guest and colleagues adapted this approach in order to
demonstrate that it was also possible to modify people’s perception of sand-
paper samples (i.e., surface properties unrelated to the body). The participants
in the latter’s studies rubbed a selection of swatches (that were hidden out of
sight in a box; see Fig. 2) with a finger by modifying the sounds made by the
interaction with the material, building on earlier work by Lederman (1979) on
auditory texture perception (see also Lemaitre and Heller, 2012).
Sound, in other words, is an integral part of our interaction with the major-
ity of materials, objects, and products. Adding that sound, or modifying it, has
been shown to influence people’s perception/behaviour in a diverse range of
situations including, in one case, in an augmented reality clothing setting in
a fashion store mirror application (Ho et al., 2013). In this study, the sounds
of different realistic jacket materials rustling were synchronized with people’s
movements when standing in front of an augmented mirror that allowed them
to try on virtual clothing visually. Meanwhile, in another project, we demon-
strated that modification of the sound made when women walked in high heels
on an augmented catwalk significantly influenced both their impressions and
emotional reactions (Tonetto et al., 2014). In particular, the sound that the
female participants heard when either of their heels touched the ground was
changed in order to convey different shoe and floor material interactions, some
louder than others (see also Bresin et al., 2010; Furfaro et al., 2015; Serafin et
al., 2011; Turchet et al., 2010).
In earlier research, Zampini et al. (2003) had modified the sound made
by an electric toothbrush (boosting or cutting the high frequency components
of the sound of the motor). Once again, this experimental manipulation was
found to influence the perception of those who used the product. And staying
with the mouth for a moment, sonic cues have also been added to modify the
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

752 C. Spence / Multisensory Research 33 (2020) 737–775
Figure 2. The experimental set-up used in Guest et al.’s (2002) study of the auditory contri-
butions to multisensory perception of surface texture. (a) This panel shows the experimenter’s
view of the automated wheel on which were mounted several different samples of sandpaper,
hidden from the participant’s view. (b) A close-up of the participant’s finger contact with the
material. The results of this study demonstrated that auditory cues exert a significant influence
over the perception of tactile texture, at least under those conditions where vision is precluded.
perceived texture of crispy foods (e.g., Masuda and Okajima, 2011; Zampini
and Spence, 2004; see Spence, 2015, for a review). Researchers in Japan have
even investigated whether the sound of the appropriate food texture can be
used to help those elderly individuals forced to eat pureed meals to enjoy their
food more (Endo et al., 2016, 2017; Fujisaki, 2020). Finally here, Spence and
Zampini (2007) modified the sound made by an aerosol spray deodorant while
in use and showed a crossmodal influence on users’ perception of the product
and its powerfulness.
6.2. Olfactory Contributions to Multisensory Shitsukan Perception
As far as olfactory cues to the multisensory perception of material/product
quality are concerned, the classic study was published by Laird (1932) almost
90 years ago. Laird reported that women’s judgements of the quality of silk
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 753
stockings depended on the scent with which the stockings had been impreg-
nated. The 250 housewives in Laird’s study were shown to prefer stockings
with a narcissus scent over those with a ‘natural’ scent, even though the stock-
ings were otherwise identical. Intriguingly, when asked for the reason behind
their preference for one pair of stockings over the others, the majority of those
questioned apparently pointed to differences in durability, sheen, or weave
(i.e., to differences in the tactile and/or visual material properties), rather than
to differences in their olfactory properties. This observation provides an early
example suggesting that we are sometimes unaware of the sensory inputs that
may actually be driving our perceptual decisions.
Demattè et al. (2006) followed up on Laird’s (1932) seminal early study,
demonstrating that olfactory cues influenced the tactile perception of fab-
ric softness using computer-controlled stimulus presentation (i.e., an eight-
channel olfactometer and a fabric carousel). The results revealed that partici-
pants rated fabric swatches as feeling significantly softer when presented with
a lemon odour than when presented with an animal-like odour instead, thus
demonstrating once again that olfactory cues can indeed modulate tactile per-
ception. Meanwhile, Churchill et al. (2009) have reported that the addition of
a variety of fragrances modified the perceived textural properties of shampoo
and hair.
At the other end of the spectrum in terms of quality perception, it is worth
drawing attention to the anecdotal reports hinting at the profound effect played
by ‘new car smell’ in modifying people’s perception of a vehicle (Moran
2000a, b, c; Van Lente and Herman, 2001). What is more, such crossmodal
influences on the perception of quality do not just affect the consumer’s per-
ception of a new vehicle, but also appear to modify people’s perception of
their own vehicles following servicing. Just take the reports from SC Gor-
don Ltd, coachbuilders of Rolls-Royce cars, who have developed their own
unique new car smell designed specifically to mimic the aromatic blend of
leather and wood of a vintage 1965 Silver Cloud model. The ‘car cologne’
is applied when new cars come in for repair. According to Hugh Hadland,
Managing Director of the company, “People say they don’t understand what
we’ve done,but that their cars come back different and better” (quote from
Spence, 2002). It seems that just one squirt of the luxury perfume is enough
to restore that sense of luxury in even the most expensive of consumer pur-
chases. Of course, such an olfactorily-inspired approach can also be used to
add value when it comes to reselling a car (Aikman, 1951; Hamilton, 1966;
Wright, 1966). And finally here, though beyond the scope of the present arti-
cle, it should also be mentioned that there is an intriguing literature examining
how the introduction/manipulation of olfactory cues can influence people’s
perception/reception of works of art (e.g., Cirrincione et al., 2014; Pursey and
Lomas, 2018).
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

754 C. Spence / Multisensory Research 33 (2020) 737–775
6.3. A Taste of Shitsukan
Research conducted over the last decade or so has demonstrated the impor-
tant role played by the metal in which a spoon has been coated on gustatory
perception (e.g., Laughlin et al., 2009, 2011; Piqueras-Fiszman et al., 2012).
Indeed, several studies have assessed people’s ability to determine the mate-
rial properties of stimuli when placed in the mouth (e.g., Howes et al., 2014;
Jacobs et al., 1998). Importantly, such effects were demonstrated by Piqueras-
Fiszman and her colleagues in the absence of any visual input (that is, the
participants were blindfolded). In the latter study, participants tasted samples
of cream that were slightly sweet, sour, bitter, salty, or plain. The samples were
tasted from spoons that were identical in terms of their size and weight, but
had been coated in zinc, copper, gold, and stainless steel. The results showed
that the taste properties of the creams were influenced by the material prop-
erties of the spoons, thus providing an example of gustatory contributions to
material perception.
Here, of course, it is interesting to note that any gustatory influence of
the spoon identified under conditions of blind tasting in Piqueras-Fiszman et
al.’s (2012) study may simply be overridden by any visual associations that
a consumer may have when interacting with such cutlery (see Spence and
Piqueras-Fiszman, 2014). That is, while one metal might improve the taste of
foods (or rather enhance a specific taste attribute, such as salty, bitter or sour)
when a taster is denied vision, it is easy to imagine how catching sight of a
gold spoon, say, might immediately set specific quality expectations (i.e., af-
fective, and possibly also sensory) and hence dominate, or modify, the ensuing
product experience (see Aldersey-Williams, 2011) (Note 4). Note here also the
fact that Harrar and Spence (2013) have documented a modest influence of the
colour of plastic spoons on the taste of yoghurt too.
7. Context and Product-Extrinsic Influences
The senses typically provide complementary information concerning the ma-
terial properties of an object or surface in the natural world. However, that is
by no means always the case (see Björkman, 1967; and Stanton and Spence,
2020, for a review). As we have seen already, the senses sometimes access
more-or-less modality-specific material properties. According to Fenko et al.
(2010b), though, the relative importance of the different senses often changes
over the various stages of product interaction/lifespan (e.g., from initial pur-
chase through to eventual disposal). What is more, either deliberately, or
accidentally, incongruent sensory cues are occasionally associated with the
impressions delivered by the different senses in augmented or virtual reality
settings (McGee et al., 2002). Relatively small differences in the impression
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 755
delivered by the different senses may well not be noticed, in part because they
may be eliminated as a result of multisensory integration (i.e., often visual
dominance). However, when the difference between the senses becomes too
large, the discrepancy may well become apparent to the observer. One of the
important unanswered questions in this research area concerns the magnitude
of the intersensory discrepancy at which crossmodal binding switches to seg-
regation (see Chen and Spence, 2017) — this is related to the causal inference
problem in multisensory perception (Körding et al., 2007).
Sensory incongruency is sometimes deliberately used by designers. Such
incongruency may either give rise to hidden or visible novelty (e.g., Ludden
and Schifferstein, 2007; Ludden et al., 2009; see also Schifferstein and Spence,
2008). Visible novelty is apparent even without the observer having to inter-
act physically with the object, whereas hidden novelty may only reveal itself
when interacting physically with the object. One example of the latter is of-
fered by those vases that look like they are made of cut glass, but which are
actually made of plastic, and so are much lighter than expected when they are
picked up (see Schifferstein and Spence, 2008). Anecdotally, hidden novelty
can lead to memorable material interactions. For instance, I vividly remem-
ber the occasion more than a decade ago when picking up what looked like
a regular vellum envelope in a Michelin-starred restaurant. I was shocked to
discover that it actually had the feel of skin (Note 5). Neuroimaging research
suggests that visual–tactile material incongruency gives rise to activation in the
precuneus (Kitada et al., 2014). As yet, however, and as noted already, there
has been relatively little neuroimaging research specifically on multisensory
incongruency in the case of material perception.
In the case of food experience, such surprising experiences of the food/dish
are often desirable and, in some cases, even increasingly expected by din-
ers, especially those visiting modernist establishments (e.g., Spence, 2017;
Vel as c o et al., 2016). There is some intriguing research on the material prop-
erties of foods that are, for example, associated with freshness (Arce-Lopera
et al., 2012; Imura et al., 2016; see also Meert et al., 2014). At the same time,
however, there is also an emerging interest in trying to change the material
properties of food by means of augmented reality (e.g., Huang et al., 2019;
Ueda et al., submitted).
7.1. Product-Extrinsic Influences
While much of the research on material perception has tended to focus on
the influence of product-intrinsic cues, i.e., cues such as roughness or shape
that can be discerned both visually and haptically (e.g., Bergmann Tiest and
Kappers, 2007), an emerging body of research has started to highlight the im-
portance of a variety of contextual cues, such as the influence of background
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

756 C. Spence / Multisensory Research 33 (2020) 737–775
music on tactile evaluation. So, for example, one study documented a signif-
icant effect of sexy music on ratings of the sexiness of the tactile stimulation
delivered by a robotic stroking device (Fritz et al., 2017). Meanwhile, other
researchers have recently reported that soft music (defined as slower-tempo,
smooth-flowing rhythms, smoothly connected legato-like notes and consonant
harmony, string instruments, and less variation in volume) modulates people’s
ratings of the softness of material (see Imschloss and Kuenhl, 2019) (Note 6).
Of course, at one level, the use of fragrance to modify material perception
might also be considered an example of a product-extrinsic cue. Indeed, the
influence of fragrance on product perception often appears to occur regardless
of whether that fragrance is perceived to have originated from the product or
material, or merely to be present as a contextual cue (see Demattè et al., 2006;
Spence, 2002). As yet, I am not aware of relevant research that has attempted
to assess whether the integration of information presented in different modal-
ities differs between the two cases (i.e., depending on whether the odorant is
treated as being product-intrinsic or merely present in the atmosphere)
At the same time, however, it would also appear that the consumer’s re-
sponse to deliberate ‘designed’ intersensory incongruency is also determined,
in part, by the context in which it is presented: think high-end design store vs.
IKEA, or Michelin-starred modernist restaurant versus canteen (Schifferstein
and Spence, 2008; see also Sundar and Noseworthy, 2016a, b). Sundar and
Noseworthy have reported on some intriguing work looking at cross-sensory
congruency/incongruency as a function of people’s perception of the pur-
ported company/brand involved. They suggest that different strategies may be
more appropriate for different brands/brand personalities. In particular, visual–
tactile incongruency in packaging design appears to be better received for
those brands that are rated as exciting (see also Littel and Orth, 2013). Sundar
and Noseworthy conducted field and laboratory studies in which their partic-
ipants were presented with products packaged in such a way that they either
looked and felt textured (crossmodally congruent), or they looked textured
but felt like something else (crossmodally incongruent). The results showed
that ‘sincere’ brands (like Hallmark, Ford, Coca-Cola) were preferred when
there was crossmodal congruency, while those brands that were judged more
exciting (BMW, Pepsi, Mountain Dew) were preferred when there was incon-
gruency between the seen and felt texture of the packaging instead.
Another kind of perceptual switch between mismatching appearance cues
occurs when products or surfaces present images on the underlying sub-
strate/surface (e.g., Childs and Henson, 2007; Jansson-Boyd and Marlow,
2007). While the image, in such cases, is normally key, there are occasions
where a deliberate attempt is made by the creator to draw the viewer’s atten-
tion to the tactile/haptic/textural attributes of such surfaces. This is captured
in the following quote from Durand (1995, p. 150) who describes how the
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 757
viewer’s attention may shift from the vision impression to touch/haptics while
inspecting a photograph, and how the very frame of reference/object of atten-
tion changes in the process: “It is interesting also to take into account those
points at which perception shifts from one regime to another — for example,
how in some photographs attention moves from the thing represented to an
awareness of texture,say the grain of the skin or the weave of foliage as they
become identified with the photographic texture itself.When that happens,it
is a real event,a moment of purely visual thought that takes place — as we
shift from a regime of pure opticality to the optical-tactile (or ‘optical-haptic’
in Alois Riegl’s terminology”(Note7).
Another area of interest in terms of the impact of contextual cues on ma-
terial perception concerns what happens when people respond to objects as a
function of where those objects are handled (e.g., museum or science labora-
tory), or what people are told about the provenance of what they are handling
(rare artefact or cheap replica). While none of the various grant applications
that I have been involved in on this very topic were ever funded, I nevertheless
still find the question to be both intriguing and important (see Chatterjee, 2008;
Pye, 2007). In somewhat conceptually-related research, it has been shown that
what people believe about other kinds of tactile stimulation (e.g., whether they
believe a skin cream to be cheap or expensive, or who they believe to be
stroking their arm) can influence both the behavioural and neural responses
to that touch that can be observed (e.g., McCabe et al., 2008).
8. Conclusions
Despite the evident, and to an extent understandable, focus on the visual as
far as research on the theme of shitsukan is concerned (see Komatsu and
Goda, 2018, for a recent authoritative review), it is important to remember
that our experience of the material qualities that we physically evaluate is typi-
cally a multisensory experience (e.g., Schifferstein and Hekkert, 2008; Spence,
2009a). And while visual cues may very well, and very often, dominate the
resulting experience and hence judgement (see Posner et al., 1976; Spence,
2011a), that does not mean that the other senses can be ignored, even though
they are undoubtedly harder to render digitally, at least given the currently
available technology. What is more, as we have seen, there are a number of
situations in which one of the non-visual senses dominates, or else modulates
multisensory quality perception (see Adams et al., 2016; Fujisaki et al., 2014;
Lederman et al., 1986).
Ultimately, it is important to remember that shitsukan is a fundamentally
multisensory construct, one that, as we have seen time-and-again throughout
this review, can be influenced by what we hear, feel, smell, and even, on oc-
casion, what we taste/feel in the mouth (see Howes et al., 2014). And true to
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

758 C. Spence / Multisensory Research 33 (2020) 737–775
the insights of ‘humaneering’ (otherwise known as ‘consumer engineering’), a
movement that emerged in North America after the Great Depression, it is the
subtle effects, like the feeling experienced by the hand on coming into contact
with the lining of the fur coat, or the arbitrary scent applied to the stockings
in Laird’s (1932) classic study, that can really make all the difference to our
evaluation of products (e.g., Sheldon and Arens, 1932; see also Cox, 1967).
At the same time, however, it is also important to bear in mind that the sensory
interactions observed when one sense is removed (such as when participants’
vision is occluded), may not be the same as those that are observed when all
of the consumers’ senses are in operation.
Another area of growing interest is to study crossmodal correspondences
in relation to material properties, both establishing underlying correspon-
dences, but then also investigating how they influence perception/evaluation
(Bliglevens et al., 2009; Decré and Cloonan, 2019; Imschloss and Kuenhl,
2019). The latter would likely seem like an area that will become increasingly
important to those working in the area of material perception in the years ahead
(see also Spence, in press). Indeed, there is growing interest in trying to convey
material properties such as pinching and scrunching fabric via mobile devices
(see Cano et al., 2017; Xiao et al., 2016). What is more, an added benefit of
enabling tactile interaction via digital channels is the likely increase in per-
ceived ownership that such tactile interactions appear to induce (Brengman et
al., 2019; see also Pantoja et al., 2020).
The last few years have seen a rapidly expanding interest in the area of
augmented retail, as for, example, in the case of online clothing/shopping
applications (e.g., de Vries et al., 2018; Flavian et al., 2019; Heller et al.,
2019; Xiao et al., 2016). There is also growing interest in the design of ma-
terial/object interaction sounds, in everything from push-buttons (Mortensen
et al., 2009) to sporting equipment (Roberts et al., 2005; see Stanton and
Spence, 2020, for a review). Interest has also grown in the area of multisensory
contributions to product packaging (Balaji et al., 2011; see also Chen et al.,
2009), and in particular, the learnt associations that may drive product percep-
tion/evaluation (De Kerpel et al., 2020; Marckhgott and Kamleitner, 2019).
All of these, then, represent promising areas for explicitly multisensory/cross-
modal shitsukan research.
Notes
1. Product designers have, though, managed to physically render many
hyper-realistic surface textures (Hara, 2004; Lupton, 2007).
2. A similar theme runs through Lipps and Lupton’s (2018) recent volume,
The senses:Design beyond vision. The latter catalogue covers many of
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 759
the design objects presented at an exhibition held at the Cooper-Hewitt
Museum in New York a few years ago illustrating material qualities that
go beyond what is ascertainable by means of visual inspection.
3. Wang and Spence (2017) have also shown that temperature can, in some
sense at least, be conveyed musically, by basing the compositions on the
crossmodal correspondences involving audition (see Wallmark, 2019).
4. Relevant here, the available research also shows how being able to see
and feel the wine glass radically modifies/enhances the tasting experience,
even though in the absence of such cues, no difference between the experi-
ence of wine presented in different wine glasses is typically observed (see
Spence, 2011c, for a review).
5. This, I suspect, much better that the sandpaper envelope in which
the bill is currently presented in London restaurant Restaurant Story
(https://restaurantstory.co.uk/), presumably corresponding to the pain of
payment.
6. Such findings can perhaps be seen as building on earlier research where
Reinoso Carvalho demonstrated that the creaminess of a luxury chocolate
could also be modified by playing one of two pieces of music (Reinoso
Carvalho et al., 2017).
7. Riegl (1901).
References
Ackerley, R., Olausson, H., Wessberg, J. and McGlone, F. (2012). Wetness perception across
body sites, Neurosci. Lett. 522, 73–77.
Ackerley, R., Saar, K., McGlone, F. and Badklund Wasling, H. (2014). Quantifying the sen-
sory and emotional perception of touch: differences between glabrous and hairy skin, Front.
Behav. Neurosci. 8, 34. DOI:10.3389/fnbeh.2014.00034.
Ackerman, J. (2016). Implications of haptic experience for product and environmental design,
in: The Psychology of Design: Creating Consumer Appeal, R. Batra, C. Seifert and D. Brei
(Eds), pp. 3–25. Routledge, London, UK.
Adams, W. J., Kerrigan, I. S. and Graf, E. W. (2016). Touch influences perceived gloss, Sci.
Rep. 6, 21866. DOI:10.1038/srep21866.
Aikman, L. (1951). Perfume, the business of illusion, Natl Geogr. Mag. 99, 531–550.
Alais, D. and Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal inte-
gration, Curr. Biol. 14, 257–262.
Aldersey-Williams, H. (2011). Periodic Tales: the Curious Lives of the Elements. Viking, Lon-
don, UK.
Altinsoy, M. E. (2008). The effect of auditory cues on the audiotactile roughness perception:
modulation frequency and sound pressure level, in: Haptic and Audio Interaction Design,
A. Pirhonen and S. Brewster (Eds), pp. 120–129. Springer, Berlin, Germany.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

760 C. Spence / Multisensory Research 33 (2020) 737–775
Altinsoy, M. E. (2020). Perceptual features of everyday push button sounds and audiotactile
interaction, Acoust. Sci. Technol. 41, 173–181.
Anderson, B. L. (2011). Visual perception of materials and surfaces, Curr. Biol. 21, R978–R983.
Aramaki, M., Besson, M., Kronland-Martinet, R. and Ystad, S. (2011). Controlling the per-
ceived material in an impact sound synthesizer, IEEE/ACM Trans. Audio Speech Lang.
Process 19, 301–314.
Arce-Lopera, C., Masuda, T., Kimura, A., Wada, Y. and Okajima, K. (2012). Luminance distri-
bution modifies the perceived freshness of strawberries, i-Perception 3, 338–355.
Arnott, S. R., Cant, J. S., Dutton, G. N. and Goodale, M. A. (2008). Crinkling and crumpling:
an auditory fMRI study of material properties, NeuroImage 43, 368–378.
Atlas, L. Y. and Wager, T. D. (2013). Expectancies and beliefs: insights from cognitive neu-
roscience, in: The Oxford Handbook of Cognitive Neuroscience, Vol. 2: the Cutting Edges,
K. N. Ochsner and S. M. Kosslyn (Eds), pp. 359–381. Oxford University Press, Oxford, UK.
Avanzini, F. and Crosato, P. (2006). Haptic-auditory rendering and perception of contact stiff-
ness, in: Haptic and Audio Interaction Design. HAID 2006,Lecture Notes in Computer
Science, Vol. 4129, D. McGookin and S. Brewster (Eds), pp. 24–35. Springer, Berlin, Ger-
many.
Balaji, M. S., Raghavan, S. and Jha, S. (2011). Role of tactile and visual inputs in product
evaluation: a multisensory perspective, Asia Pac. J. Mark. Logist. 23, 513–530.
Ballesteros, S., Reales, J. M., de León, L. P. and Garcia, B. (2005). The perception of ecological
textures by touch: does the perceptual space change under bimodal visual and haptic explo-
ration?, in: Proceedings of First Joint Eurohaptics Conference and Symposium on Haptic
Interfaces for Virtual Environment and Teleoperator Systems, 2005. World Haptics 2005,
pp. 635–638. IEEE Computer Society, Los Alamitos, CA, USA.
Baumgartner, E., Wiebel, C. B. and Gegenfurtner, K. R. (2013). Visual and haptic representa-
tions of material properties, Multisens. Res. 26, 429–455.
Berger, G., Katz, H. and Petutshnigg, A. J. (2006). What consumers feel and prefer: haptic
perception of various wood flooring surfaces, For. Prod. J. 56, 42–47.
Bergmann Tiest, W. M. (2010). Tactual perception of material properties, Vision Res. 50, 2775–
2782.
Bergmann Tiest, W. M. (2015). Tactual perception of liquid material properties, Vision Res. 109,
178–184.
Bergmann Tiest, W. M. and Kappers, A. M. L. (2006). Analysis of haptic perception of materials
by multidimensional scaling and physical measurements of roughness and compressibility,
Acta Psychol. 121, 1–20.
Bergmann Tiest, W. M. and Kappers, A. M. L. (2007). Haptic and visual perception of rough-
ness, Acta Psychol. 124, 177–189.
Bergmann Tiest, W. M. and Kappers, A. M. L. (2009). Tactile perception of thermal diffusivity,
Atten. Percept. Psychophys. 71, 481–489.
Bhatta, S. R., Tiippana, K., Vahtikari, K., Hughes, M. and Kyttä, M. (2017). Sensory and
emotional perception of wooden surfaces through fingertip touch, Front. Psychol. 8, 367.
DOI:10.3389/fpsyg.2017.00367.
Bicchi, A., Buss, M., Ernst, M. O. and Peer, A. (Eds) (2008). The Sense of Touch and Its Ren-
dering. Springer, Berlin, Germany.
Binninger, A.-S. (2017). Perception of naturalness of food packaging and its role in consumer
product evaluation, J. Food Prod. Mark. 23, 251–266.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 761
Binns, H. (1926). The discrimination of wool fabrics by the sense of touch, Br. J. Psychol. 16,
237–247.
Binns, H. (1934). A visual and tactual analysis of typical Bradford wool tops, J. Text. Inst. 25,
T331–T354.
Binns, H. (1937). Visual and tactual ‘judgement’ as illustrated in a practical experiment, Br. J.
Psychol. 27, 404–410.
Björk, E. A. (1985). The perceived quality of natural sounds, Acustica 57, 185–188.
Björkman, M. (1967). Relations between intra-modal and cross-modal matching, Scand. J. Psy-
chol. 8, 65–76.
Blijlevens, J., Creusen, M. E. H. and Schoormans, J. P. L. (2009). How consumers perceive
product appearance: the identification of three product appearance attributes, Int. J. Des. 3,
27–35.
Bone, P. F. and Jantrania, S. (1992). Olfaction as a cue for product quality, Mark. Lett. 3, 289–
296.
Bosmans, A. (2006). Scents and sensibility: when do (in)congruent ambient scents influence
product evaluations?, J. Mark. 70, 32–43.
Brengman, M., Willems, K. and Van Kerrebroeck, H. (2019). Can’t touch this: the impact of
augmented reality versus touch and non-touch interfaces on perceived ownership, Virt . Real.
23, 269–280.
Bresin, R., deWitt, A., Papetti, S., Civolani, M. and Fontana, F. (2010). Expressive sonification
of footstep sounds, in: Proceedings of ISon 2010, 3rd Interactive Sonification Workshop
KTH, Stockholm, Sweden, April 7, pp. 51–54.
Brown, S., Gao, X., Tisdelle, L., Eickhoff, S. B. and Liotti, M. (2011). Naturalizing aesthetics:
brain areas for aesthetic appraisal across sensory modalities, NeuroImage 58, 250–258.
Buckingham, G., Cant, J. S. and Goodale, M. A. (2009). Living in a material world: how visual
cues to material properties affect the way that we lift objects and perceive their weight,
J. Neurophysiol. 102, 3111–3118.
Byron, E. (2012). The search for sweet sounds that sell: Household products’ clicks and hums
are no accident; Light piano music when the dishwasher is done? Wall Street J. October 23rd.
http://online.wsj.com/article/SB10001424052970203406404578074671598804116.html?
mod=googlenews_wsj#articleTabs%3Darticle.
Cano, M. B., Perry, P., Ashman, R. and Waite, K. (2017). The influence of image interactivity
upon user engagement when using mobile touch screens, Comput. Hum. Behav. 77, 406–
412.
Cant, J. S. and Goodale, M. A. (2007). Attention to form or surface properties modulates dif-
ferent regions of human occipitotemporal cortex, Cereb. Cortex 17, 713–731.
Chadwick, A. C. and Kentridge, R. W. (2015). The perception of gloss: a review, Vision Res.
109(B), 221–235.
Chadwick, A. C., Heywood, C. A., Smithson, H. E. and Kentridge, R. W. (2017). Translucence
perception is not dependent on cortical areas critical for processing colour or texture, Neu-
ropsychologia 128, 209–214. DOI:10.1016/j.neuropsychologia.2017.11.009.
Chang, H. J., Song, J., Yeo, C. and Kim, J. (2015). Exploring factors influencing perceived
quality on sportswear fabric, in: International Textile and Apparel Association (ITAA) An-
nual Conference Proceedings, Abstract 18.
Chatterjee, H. (2008). Touch in Museums: Policy and Practice in Object Handling. Berg Publi-
cations, Oxford UK.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

762 C. Spence / Multisensory Research 33 (2020) 737–775
Chen, X., Shao, F., Barnes, C., Childs, T. and Henson, B. (2009). Exploring relationships be-
tween touch perception and surface physical properties, Int. J. Des. 3, 67–76.
Chen, Y.-C. and Spence, C. (2017). Assessing the role of the ‘unity assumption’ on multisensory
integration: a review, Front. Psychol. 8, 445. DOI:10.3389/fpsyg.2017.00445.
Childers, T. L. and Peck, J. (2010). Informational and affective influences of haptics on product
evaluation: is what I say how I feel?, in: Sensory Marketing: Research on the Sensuality of
Products, A. Krishna (Ed.), pp. 63–72. Routledge, New York, NY, USA.
Childs, T. H. C. and Henson, B. (2007). Human tactile perception of screen-printed surfaces:
self-report and contact mechanics experiments, J. Eng. Tribol. 221, 427–441.
Churchill, A., Meyners, M., Griffiths, L. and Bailey, P. (2009). The cross-modal effect of fra-
grance in shampoo: modifying the perceived feel of both product and hair during and after
washing, Food Qual. Prefer. 20, 320–328.
Cirrincione, A., Estes, Z. and Carù, A. (2014). The effect of ambient scent on the experience of
art: not as good as it smells, Psychol. Mark. 31, 615–627.
Citrin, A. V., Stem, D. E., Spangenberg, E. R. and Clark, M. J. (2003). Consumer need for tactile
input: an Internet retailing challenge, J. Bus. Res. 56, 915–922.
Cox, D. F. (1967). The sorting rule model of the consumer product evaluation process, in:
Risk Taking and Information Handling in Consumer Behaviour, D. F. Cox (Ed.), pp. 324–
369. Division of Research, Graduate School of Business Administration, Harvard University,
Boston, MA, USA.
Culbert, S. S. and Stellwagen, W. T. (1963). Tactual discrimination of textures, Percept. Mot.
Skills 16, 545–552.
De Kerpel, L., Kobuszewski Volles, B. and Van Kerckhove, A. (2020). Fats are glossy, but does
glossiness imply fatness? The influence of packaging glossiness on food perceptions, Foods
2020 9, 90. DOI:10.3390/foods9010090.
de Vries, R., Jager, G., Tijssen, I. and Zandstra, E. H. (2018). Shopping for products in a virtual
world: why haptics and visuals are equally important in shaping consumer perceptions and
attitudes, Food Qual. Prefer. 66, 64–75.
Decré, G. B. and Cloonan, C. (2019). A touch of gloss: haptic perception of packaging and
consumers’ reactions, J. Prod. Brand Manag. 28, 117–132.
Demattè, M. L., Sanabria, D., Sugarman, R. and Spence, C. (2006). Cross-modal interactions
between olfaction and touch, Chem. Senses 31, 291–300.
Demattè, M. L., Österbauer, R. and Spence, C. (2007a). Olfactory cues modulate facial attrac-
tiveness, Chem. Senses 32, 603–610.
Demattè, M. L., Sanabria, D. and Spence, C. (2007b). Olfactory–tactile compatibility effects
demonstrated using the Implicit Association Test, Acta Psychol. 124, 332–343.
Deroy, O., Fernandez-Prieto, I., Navarra, J. and Spence, C. (2018). Unravelling the paradox of
spatial pitch, in: Spatial Biases in Perception and Cognition, T. L. Hubbard (Ed.), pp. 77–93.
Cambridge University Press, Cambridge, UK.
Drucker, S. M. (1988). Texture from touch, in: Natural Computation,W.Richards(Ed.),
pp. 422–429. MIT Press, Cambridge, MA, USA.
Durand, R. (1995). How to see (photographically), in: Fugitive Images: from Photography to
Vid eo, P. Petro (Ed.), pp. 141–151. Indiana University Press, Bloomington, IN, USA.
Eck, J., Kaas, A. L. and Goebel, R. (2013). Crossmodal interactions of haptic and visual texture
information in early sensory cortex, NeuroImage 75, 123–135.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 763
Endo, H., Ino, S. and Fujisaki, W. (2016). The effect of a crunchy pseudo-chewing sound on
perceived texture of softened foods, Physiol. Behav. 167, 324–331.
Endo, H., Ino, S. and Fujisaki, W. (2017). Texture-dependent effects of pseudo-chewing sound
on perceived food texture and evoked feelings in response to nursing care foods, Appetite
116, 493–501.
Ernst, M. O. (2007). Learning to integrate arbitrary signals from vision and touch, J. Vis. 7,7.
DOI:10.1167/7.5.7.
Ernst, M. O. and Banks, M. S. (2002). Humans integrate visual and haptic information in a
statistically optimal fashion, Nature 415, 429–433.
Essick, G. K., McGlone, F., Dancer, C., Fabricant, D., Ragin, Y., Phillips, N., Jones, T. and
Guest, S. (2010). Quantitative assessment of pleasant touch, Neurosci. Biobehav. Rev. 34,
192–203.
Etzi, R., Ferrise, F., Bordegoni, M., Zampini, M. and Gallace, A. (2018). The effect of visual
and auditory information on the perception of pleasantness and roughness of virtual surfaces,
Multisens. Res. 31, 501–522.
Etzi, R., Spence, C. and Gallace, A. (2014). Textures that we like to touch: an experimental
study of aesthetic preferences for tactile stimuli, Consc. Cogn. 29, 178–188.
Etzi, R., Spence, C., Zampini, M. and Gallace, A. (2016). When sandpaper is ‘Kiki’ and satin is
‘Bouba’: an exploration of the associations between words, emotional states, and the tactile
attributes of everyday materials, Multisens. Res. 29, 133–155.
Felleman, D. J. and Van Essen, D. C. (1991). Distributed hierarchical processing in the primate
cerebral cortex, Cereb. Cortex 1, 1–47.
Fenko, A., Schifferstein, H. N. J. and Hekkert, P. (2010a). Looking hot or feeling hot: what
determines the product experience of warmth?, Mater. Des. 31, 1325–1331.
Fenko, A., Schifferstein, H. N. J. and Hekkert, P. (2010b). Shifts in sensory dominance between
various stages of user–product interactions, Appl. Ergon. 41, 34–40.
Fenko, A., Schifferstein, H. N. J. and Hekkert, P. (2011). Noisy products: does appearance
matter?, Int. J. Des. 5, 77–87.
Flavián, C., Ibáñez-Sánchez, S. and Orús, C. (2019). The impact of virtual, augmented and
mixed reality technologies on the customer experience, J. Bus. Res. 100, 547–560.
Fleming, R. W. (2014). Visual perception of materials and their properties, Vision Res. 94, 62–
75.
Fraser, A. (1892). The psychological foundation of natural realism, Am.J.Psychol.4, 429–450.
Fritz, T. H., Brummerloh, B., Urquijo, M., Wegner, K.,Reimer, E., Gutekunst, S., Schneider, L.,
Smallwood, J. and Villringer, A. (2017). Blame it on the bossa nova: transfer of perceived
sexiness from music to touch, J. Exp. Psychol. Gen. 146, 1360–1365.
Fujisaki, W. (2020). Multisensory Shitsukan perception, Acoust. Sci. Technol. 41, 189–195.
Fujisaki, W., Goda, N., Motoyoshi, I., Komatsu, H. and Nishida, S. (2014). Audiovisual inte-
gration in the human perception of materials, J. Vis. 14, 12. DOI:10.1167/14.4.12.
Fujisaki, W., Tokita, M. and Kariya, K. (2015). Perception of the material properties of wood
based on vision, audition, and touch, Vision Res. 109, 185–200.
Furfaro, E., Bevilacqua, F., Berthouze, N. and Tajadura-Jiménez, A. (2015). Sonification of
virtual and real surface tapping: evaluation of behavior changes, surface perception and emo-
tional indices, IEEE MultiMedia, 1–26. DOI:10.1109/MMUL.2015.30.
Gallace, A. and Spence, C. (2011). Tactile aesthetics: towards a definition of its characteristics
and neural correlates, Soc. Semiot. 21, 569–589.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

764 C. Spence / Multisensory Research 33 (2020) 737–775
Gallace, A. and Spence, C. (2014). In Touch With the Future: the Sense of Touch From Cognitive
Neuroscience to Virtual Reality. Oxford University Press, Oxford, UK.
Gallace, A., Ngo, M. K., Sulaitis, J. and Spence, C. (2012). Multisensory presence in virtual
reality: possibilities & limitations, in: Multiple Sensorial Media Advances and Applications:
New Developments in MulSeMedia, G. Ghinea, F. Andres and S. Gulliver (Eds), pp. 1–38.
IGI Global, Hershey, PA, USA.
Gau, R. and Noppeney, U. (2016). How prior expectations shape multisensory perception, Neu-
roImage 124, 876–886.
Gerdes, A. B. M., Wieser, M. J. and Alpers, G. W. (2014). Emotional pictures and sounds: a
review of multimodal interactions of emotion cues in multiple domains, Front. Psychol. 5,
1351. DOI:10.3389/fpsyg.2014.01351.
Giordano, B. L. and McAdams, S. (2006). Material identification of real impact sounds: effect
of size variation in steel, glass, wood, and plexiglass plates, J. Acoust. Soc. Am. 119, 1171–
1181.
Goda, N., Yokoi, I., Tachibana, A., Minamimoto, T. and Komatsu, H. (2016). Crossmodal as-
sociation of visual and haptic material properties of objects in the monkey ventral visual
cortex, Curr. Biol. 26, 928–934.
Goebl, W., Bresin, R. and Fujinaga, I. (2014). Perception of touch quality in piano tones,
J. Acoust. Soc. Am. 136, 2839–2850.
Guest, S. and Spence, C. (2003a). Tactile dominance in speeded discrimination of textures, Exp.
Brain Res. 150, 201–207.
Guest, S. and Spence, C. (2003b). What role does multisensory integration play in the visuotac-
tile perception of texture?, Int. J. Psychophysiol. 50, 63–80.
Guest, S., Catmur, C., Lloyd, D. and Spence, C. (2002). Audiotactile interactions in roughness
perception, Exp. Brain Res. 146, 161–171.
Haase, J. and Wiedmann, K.-P. (2018). The sensory perception item set (SPI): an exploratory
effort to develop a holistic scale for sensory marketing, Psychol. Mark. 35, 727–739.
Hagtvedt, H. and Brasel, S. A. (2016). Crossmodal communication: sound frequency influences
consumer responses to color lightness, J. Mark. Res. 53, 551–562.
Hamilton, A. (1966). What science is learning about smell, Sci. Dig. 55, 81–84.
Han, Y. (2018). All that glitters is not gold: packaging glossiness, attention, and trustworthiness,
in: E – European Advances in Consumer Research, Vol. 11, M. Geuens, M. Pandelaere, M. T.
Pham and I. Vermeir (Eds), pp. 81–82. Association for Consumer Research, Duluth, MN,
USA.
Hara, K. (2004). Haptic: awakening the Senses. Exhibition catalogue. Opendoors Books.
Harrar, V. and Spence, C. (2013). The taste of cutlery: how the taste of food is affected by the
weight, size, shape, and colour of the cutlery used to eat it, Flavour 2, 21. DOI:10.1186/
2044-7248-2-21.
Heilig, M. L. (1992). El cine del futuro: the cinema of the future, Presence (Camb.) 1, 279–294.
Heller, J., Chylinski, M., de Ruyter, K., Mahr, D. and Keeling, D. I. (2019). Touching the
untouchable: exploring multi-sensory augmented reality in the context of online retailing,
J. Retail. 95, 219–234.
Heller, M. A. (1982). Visual and tactual texture perception: intersensory cooperation, Percept.
Psychophys. 31, 339–344.
Hershberger, W. A. and Misceo, G. (1996). Touch dominates haptic estimates of discordant
visual-haptic size, Percept. Psychophys. 58, 1124–1132.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 765
Ho, C., Jones, R., King, S., Murray, L. and Spence, C. (2013). Multisensory augmented reality in
the context of a retail clothing application, in: (((ABA))) Audio Branding Academy Yearbook
2012/2013, K. Bronner, R. Hirt and C. Ringe (Eds), pp. 167–175. Nomos, Baden-Baden,
Germany.
Ho, Y.-X., Landy, M. S. and Maloney, L. T. (2008). Conjoint measurement of gloss and surface
texture, Psychol. Sci. 19, 196–204.
Hollins, M., Faldowski, R., Rao, S. and Young, F. (1993). Perceptual dimensions of tactile
surface texture: a multidimensional scaling analysis, Percept. Psychophys. 54, 697–705.
Howes, P. D., Wongsriruksa, S., Laughlin, Z., Witchel, H. J. and Miodownik, M. (2014). The
perception of materials though oral sensation, PLoS One 9, e105035. DOI:10.1371/journal.
pone.0105035.
Hsiao, S. S. (1998). Similarities between touch and vision, Adv. Psychol. 127, 131–165.
Huang, F., Huang, J. and Wan, X. (2019). Influence of virtual color on taste: multisensory
integration between virtual and real worlds, Comput. Hum. Behav. 95, 168–174.
Hunter, R. S. (1975). The Measurement of Appearance. Wiley-Interscience, Hoboken, NJ, USA.
Hutmacher, F. (2019). Why is there so much more research on vision than on any other sensory
modality?, Front. Psychol. 10, 2246. DOI:10.3389/fpsyg.2019.02246.
Imschloss, M. and Kuenhl, C. (2019). Feel the music! Exploring the cross-modal correspon-
dence between music and haptic perceptions of softness, J. Retail. 95, 158–169.
Imura, T., Masuda, T., Wada, Y., Tomonaga, M. and Okajima, K. (2016). Chimpanzees can
visually perceive differences in the freshness of foods, Sci. Rep. 6, 34685. DOI:10.1038/
srep34685.
Jacobs, R., Serhal, C. B. and van Steenberghe, D. (1998). Oral stereognosis: a review of the
literature, Clin. Oral Invest. 2, 3–10.
Jansson-Boyd, C. and Marlow, N. (2007). Not only in the eye of the beholder: tactile information
can affect aesthetic evaluation, Psychol. Aesthet. Creat. Arts 1, 170–173.
Jones, B. and O’Neil, S. (1985). Combining vision and touch in texture perception, Percept.
Psychophys. 37, 66–72.
Jones, L. A. and Ho, H.-N. (2008). Warm or cool, large or small? The challenge of thermal
displays, IEEE Trans. Haptics 1, 53–70.
Jostmann, N. B., Lakens, D. and Schubert, T. W. (2009). Weight as an embodiment of impor-
tance, Psychol. Sci. 20, 1169–1174.
Jousmäki, V. and Hari, R. (1998). Parchment-skin illusion: sound-biased touch, Curr. Biol. 8,
R190.
Kahrimanovic, M., Bergmann Tiest, W. M. and Kappers, A. M. L. (2010). Seeing and feeling
volumes: the influence of shape on volume perception, Acta Psychol. 134, 385–390.
Kampfer, K., Leischnig, A., Ivens, B. S. and Spence, C. (2017). Touch-flavor transference:
assessing the effect of packaging weight on gustatory evaluations, desire for food and bever-
ages, and willingness to pay, PLoS ONE 12, e0186121. DOI:10.1371/journal.pone.0186121.
Kanaya, S., Kariya, K. and Fujisaki, W. (2016). Cross-modal correspondence among vision,
audition, and touch in natural objects: an investigation of the perceptual properties of wood,
Perception 45, 1099–1114.
Kanie, K., Kurono, Y., Nagata, Y. and Koori, I. (1987). Vehicle door design based on sounds
and noise control, Jpn. Soc. Automot. Eng. Rev. 8, 32–37.
Karlsson, M. and Velasco, A. V. (2007). Designing for the tactile sense: investigating the relation
between surface properties, perceptions and preferences, CoDesign 3(Suppl. 1), 123–133.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

766 C. Spence / Multisensory Research 33 (2020) 737–775
Kergoat, M., Giboreau, A., Nicod, H., Faye, P., Diaz, E., Beetschen, M.-A. and Meyer, T. (2012).
Consumer preference for tactile softness: a question of affect intensity?, J. Sens. Stud. 27,
232–246.
Kim, S.-C., Kyung, K.-U. and Kwon, D.-S. (2007). The effect of sound on haptic perception,
in: IEEE Proceedings of the 2nd Joint EuroHaptics Conference and Symposium on Haptic
Interfaces for Virtual Environment and Teleoperator Systems (WHC 07), pp. 354–360.
Kitada, R., Sasaki, A. T., Okamoto, Y., Kochiyama, T. and Sadato, N. (2014). Role of the pre-
cuneus in the detection of incongruency between tactile and visual texture information: a
functional MRI study, Neuropsychologia 64, 252–262.
Klatzky, R. L. and Lederman, S. J. (2010). Multisensory texture perception, in: Multisensory
Object Perception in the Primate Brain, J. Kaiser and M. J. Naumer (Eds), pp. 211–230.
Springer, New York, NY, USA.
Klatzky, R. L., Lederman, S. J. and Metzger, V. A. (1985). Identifying objects by touch: an
‘expert system’, Percept. Psychophys. 37, 299–302.
Klatzky, R. L., Lederman, S. J. and Matula, D. E. (1993). Haptic exploration in the presence of
vision, J. Exp. Psychol. Hum. Percept. Perform. 19, 726–743.
Klatzky, R. L., Pai, D. K. and Krotkov, E. P. (2000). Perception of material from contact sounds,
Presence (Camb.) 9, 399–410.
Komatsu, H. and Goda, N. (2018). Neural mechanisms of material perception: quest on shit-
sukan, Neuroscience 392, 329–347.
Komatsuzaki, T., Han, J. and Uchida, H. (2016). Approach for combining physical properties
and sensibility for pleasant beverage can-opening sound, Appl. Acoust. 103, 64–70.
Körding, K. P., Beierholm, U., Ma, W. J., Quartz, S., Tenenbaum, J. B. and Shams, L. (2007).
Causal inference in multisensory perception, PLoS ONE 2, e943. DOI:10.1371/journal.pone.
0000943.
Krishna, A. and Morrin, M. (2008). Does touch affect taste? The perceptual transfer of product
container haptic cues, J. Consum. Res. 34, 807–818.
Krishna, A., Elder, R. S. and Caldara, C. (2010). Feminine to smell but masculine to touch?
Multisensory congruence and its effect on the aesthetic experience, J. Consum. Psychol. 20,
410–418.
Labbe, D., Pineau, N. and Martin, N. (2013). Food expected naturalness: impact of visual, tactile
and auditory packaging material properties and role of perceptual interactions, Food Qual.
Prefer. 27, 170–178.
LaBonte, D. A. (2009). Shiny Objects Marketing: Using Simple Human Instincts to Make Your
Brand Irresistible. John Wiley & Sons, Hoboken, NJ, USA.
Lacey, S. and Sathian, K. (2014). Visuo-haptic multisensory object recognition, categorization,
and representation, Front. Psychol. 5, 730. DOI:10.3389/fpsyg.2014.00730.
Lageat, T., Czellar, S. and Laurent, G. (2003). Engineering hedonic attributes to generate per-
ceptions of luxury: consumer perception of an everyday sound, Mark. Lett. 14, 97–109.
Laird, D. A. (1932). How the consumer estimates quality by subconscious sensory impressions,
J. Appl. Psychol. 16, 241–246.
Laughlin, Z., Conreen, M., Witchel, H. and Miodownik, M. A. (2009). The taste of materials:
spoons, in: Proceedings of the MINET Conference: Measurement, Sensation and Cognition,
pp. 127–128. National Physical Laboratories, Teddington, UK.
Laughlin, Z., Conreen, M., Witchel, H. J. and Miodownik, M. A. (2011). The use of standard
electrode potentials to predict the taste of solid metals, Food Qual. Pref. 22, 628–637.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 767
Lederman, S. J. (1979). Auditory texture perception, Perception 8, 93–103.
Lederman, S. J. (1982). The perception of texture by touch, in: Tactual Perception: a Source-
book, W. Schiff and E. Foulke (Eds), pp. 130–167. Cambridge University Press, Cambridge,
UK.
Lederman, S. J. and Abbott, S. G. (1981). Texture perception: studies of intersensory organisa-
tion using a discrepancy paradigm and visual versus tactual psychophysics, J. Exp. Psychol.
Hum. Percept. Perform. 7, 902–915.
Lederman, S. J. and Klatzky, R. L. (2004). Multisensory texture perception, in: The Handbook
of Multisensory Processes, G. A. Calvert, C. Spence and B. E. Stein (Eds), pp. 107–123.
MIT Press, Cambridge, MA, USA.
Lederman, S. J., Thorne, G. and Jones, B. (1986). Perception of texture by vision and touch:
multidimensionality and intersensory integration, J. Exp. Psychol. Hum. Percept. Perform.
12, 169–180.
Lemaitre, G. and Heller, L. M. (2012). Auditory perception of material is fragile while action is
strikingly robust, J. Acoust. Soc. Am. 131, 1337–1348.
Leswing, K. (2016). Apple Ceo Tim Cook thinks augmented reality will be as im-
portant as ‘eating three meals a day’, Business Insider, October 3rd. Retrieved
from https://www.businessinsider.nl/apple-ceo-tim-cook-explains-augmented-reality-
2016-10/?international=true&r=US.
Levin, M. D. (1993). Modernity and the Hegemony of Vision. University of California Press,
Berkeley, CA, USA.
Li, W., Moallem, I., Paller, K. A. and Gottfried, J. A. (2007). Subliminal smells can guide social
preferences, Psychol. Sci. 18, 1044–1049.
Lin, M. C. and Otaduy, M. A. (Eds) (2008). Haptic Rendering: Foundations, Algorithms, and
Applications. AK Peters, New York, NY, USA.
Lindauer, M. S. (1986). Seeing and touching aesthetic objects: II. Descriptions, Bull. Psychon.
Soc. 24, 125–126.
Lindauer, M. S., Stergiou, E. A. and Penn, D. L. (1986). Seeing and touching aesthetic objects:
I. Judgments, Bull. Psychon. Soc. 24, 121–124.
Lipps, A. and Lupton, E. (2018). The Senses: Design Beyond Vision. Princeton Architectural
Press, Hudson, NY, USA.
Littel, S. and Orth, U. R. (2013). Effects of package visuals and haptics on brand evaluations,
Eur. J. Mark. 47, 198–217.
Löken, L. S., Wessberg, J., Morrison, I., McGlone, F. and Olausson, H. (2009). Coding of
pleasant touch by unmyelinated afferents in humans, Nat. Neurosci. 12, 547–548.
Ludden, G. D. S. and Schifferstein, H. N. J. (2007). Effects of visual–auditory incongruity on
product expression and surprise, Int. J. Des. 1, 29–39.
Ludden, G. D. S. and van Rompay, T. J. L. (2015). How does it feel? Exploring touch on
different levels of product experience, J. Eng. Des. 26, 157–168.
Ludden, G. D. S., Schifferstein, H. N. J. and Hekkert, P. (2009). Visual–tactual incongruities in
products as sources of surprise, Empir Stud. Arts 27, 61–87.
Lupton, E. (2002). Skin: Surface, Substance, and Design. Princeton Architectural Press, New
York, NY, USA.
Marckhgott, E. and Kamleitner, B. (2019). Matte matters: when matte packaging increases per-
ceptions of food naturalness, Mark. Lett. 30, 167–178.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

768 C. Spence / Multisensory Research 33 (2020) 737–775
Masuda, M. and Okajima, K. (2011). Added mastication sound affects food texture and pleas-
antness, i-Perception 2, 949.
McCabe, C., Rolls, E. T., Bilderbeck, A. and McGlone, F. (2008). Cognitive influences on the
affective representation of touch and the sight of touch in the human brain, Soc. Cogn. Affect.
Neurosci. 3, 97–108.
McDermott, J. H. and Simoncelli, E. P. (2011). Sound texture perception via statistics of the
auditory periphery: evidence from sound synthesis, Neuron 71, 926–940.
McGee, M. R., Gray, P. and Brewster, S. (2002). Mixed feelings: Multimodal perception of vir-
tual roughness, in: Proceedings of the International Conference of EuroHaptics Edinburgh,
pp. 47–52.
McGlone, F., Österbauer, R. A., Demattè, L. M. and Spence, C. (2013). The crossmodal influ-
ence of odor hedonics on facial attractiveness: behavioural and fMRI measures, in: Func-
tional Brain Mapping and the Endeavor to Understand the Working Brain, F. Signorelli and
D. Chirchiglia (Eds), pp. 209–225. InTech Publications, Rijeka, Croatia.
McGlone, F. and Spence, C. (2010). The cutaneous senses: touch, temperature, pain/itch, and
pleasure, Neurosci. Biobehav. Rev. 34, 145–147.
Meert, K., Pandelaere, M. and Patrick, V. M. (2014). Taking a shine to it: how the preference
for glossy stems from an innate need for water, J. Consum. Psychol. 24, 195–206.
Meijer, D., Veseliˇ
c, S., Calafiore, C. and Noppeney, U. (2019). Integration of audiovisual spatial
signals is not consistent with maximum likelihood estimation, Cortex 119, 74–88.
Michel, C., Velasco, C. and Spence, C. (2015). Cutlery matters: heavy cutlery enhances diners’
enjoyment of the food served in a realistic dining environment, Flavour 4, 26. DOI:10.1186/
s13411-015-0036-y.
Mirzoeff, N. (1999). An Introduction to Visual Culture. Routledge, London, UK.
Montignies, F., Nosulenko, V. and Parizet, E. (2010). Empirical identification of perceptual
criteria for customer-centred design. Focus on the sound of tapping on the dashboard when
exploring a car, Int. J. Ind. Ergon. 40, 592–603.
Moran, T. (2000a). Ah, the aroma of a just-baked sedan, N. Y. Times May 14th, 40.
Moran, T. (2000b). Sniffing car parts: yes, the job stinks, N. Y. Times May 14th, 40.
Moran, T. (2000c). The mysterious human sense of smell: so primitive and so powerful, N. Y.
Times May 14th, 40.
Mortensen, D. H., Bech, S., Begault, D. R. and Adelstein, B. D. (2009). The relative importance
of visual, auditory, and haptic information for the user’s experience of mechanical switches,
Perception 38, 1560–1571.
Murakoshi, T., Masuda, T., Utsumi, K., Tsubota, K. and Wada, Y. (2013). Glossiness and per-
ishable food quality: visual freshness, judgment of fish eyes based on luminance distribution,
PLoS ONE 8, e58994. DOI:10.1371/journal.pone.0058994.
Nagai, T., Matsushima, T., Koida, K., Tani, Y., Kitazaki, M. and Nakauchi, S. (2015). Tempo-
ral properties of material categorization and material rating: visual vs non-visual material
features, Vision Res. 115, 259–270.
Nagano, H., Okamoto, S. and Yamada, Y. (2014). Haptic invitation of textures: perceptually
prominent properties of materials determine human touch motions, IEEE Trans. Haptics 7,
345–355.
Nikolaidou, I. (2011). Communicating naturalness through packaging design, in: From Floating
Wheelchairs to Mobile Car Parks, P. M. A. Desmet and H. N. J. Schifferstein (Eds), pp. 74–
79. Eleven International, The Hague, NL.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 769
Norman, J. F., Phillips, F., Holmin, J. S., Norman, H. F., Beers, A. M., Boswell, A. M., Cheese-
man, J. R., Stgethen, A. G. and Ronning, C. (2012). Solid shape discrimination from vision
and haptics: natural objects (Capsicum annuum) and Gibson’s ‘feelies’, Exp. Brain Res. 222,
321–332.
Okamoto, S., Nagano, H. and Yamada, Y. (2013). Psychophysical dimensions of tactile percep-
tion of textures, IEEE Trans. Haptics 6, 81–93.
Overmars, S. and Poels, K. (2015). A touching experience: designing for touch sensations in
online retail environments, Int. J. Des. 9, 17–31.
Overvliet, K. E. and Soto-Faraco, S. (2011). I can’t believe this isn’t wood! An investigation in
the perception of naturalness, Acta Psychol. 136, 95–111.
Overvliet, K. E., Karana, E. and Soto-Faraco, S. (2016). Perception of naturalness in textiles,
Mater. Des. 90, 1192–1199.
Owens, A., Isola, P., McDermott, J., Torralba, A., Adelson, E. H. and Freeman, W. T. (2016).
Visually indicated sounds, in: Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), pp. 2405–2413. IEEE.
Özcan, E. and van Egmond, R. (2012). Basic semantics of product sounds, Int. J. Des. 6, 41–54.
Pantoja, F., Borges, A., Rossi, P. and Yamim, A. P. (2020). If I touch it, I will like it! The role
of tactile inputs on gustatory perceptions of food items, J. Retail. Consum. Serv. 53, 101958.
DOI:10.1016/j.jretconser.2019.101958.
Parise, C. V., Knorre, K. and Ernst, M. O. (2014). Natural auditory scene statistics shapes human
spatial hearing, Proc. Natl Acad. Sci. USA 111, 6104–6108.
Parisi, D. (2018). Archaeologies of Touch: Interfacing With Haptics From Electricity to Com-
puting. University of Minnesota Press, Minneapolis, MN, USA.
Parthasarathy, S. and Chhapgar, A. F. (1955). Sound absorption in liquids in relationship to their
physical properties: viscosity and specific heats, Ann. Phys. 451, 297–303.
Peck, J. and Childers, T. L. (2003a). Individual differences in haptic information processing:
the ‘need for touch’ scale, J. Consum. Res. 30, 430–442.
Peck, J. and Childers, T. L. (2003b). To have and to hold: the influence of haptic information on
product judgments, J. Mark. 67, 35–48.
Peck, J. and Childers, T. L. (2008). Effects of sensory factors on consumer behavior: if it tastes,
smells, sounds, and feels like a duck, then it must be a...,in:Marketing and Consumer
Psychology Series: Vol. 4,Handbook of Consumer Psychology,C.P.Haugtvedt,P.M.Herr
and F. R. Kardes (Eds), pp. 193–219. Psychology Press, New York, NY, USA.
Peeva, D., Baird, B., Izmirli, O. and Blevins, D. (2004). Haptic and sound correlations: pitch,
loudness and texture, in: Proceedings of the Eighth International Conference on Information
Visualization (IV 2004), E. Banissi (Ed.), pp. 659–664. IEEE Computer Society, London,
UK.
Pérez-Bellido, A., Barnes, K. A., Crommett, L. E. and Yau, J. M. (2018). Auditory frequency
representations in human somatosensory cortex, Cereb. Cortex 28, 3908–3921.
Peters, M. A. K., Balzer, J. and Shams, L. (2015). Smaller =denser, and the brain knows
it: natural statistics of object density shape weight expectations, PLoS ONE 10, e0119794.
DOI:10.1371/journal.pone.0119794.
Peters, M. A. K., Zhang, L. Q. and Shams, L. (2018). The material-weight illusion is a Bayes-
optimal percept under competing density priors, PeerJ 6, e5760. DOI:10.7717/peerj.5760.
Petit, O., Velasco, C. and Spence, C. (2019). Digital sensory marketing: integrating new tech-
nologies into multisensory online experience, J. Interact. Mark. 45, 42–61.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

770 C. Spence / Multisensory Research 33 (2020) 737–775
Philippe, F., Schacher, L., Adolphe, D. C. and Dacremont, C. (2004). Tactile feeling: sensory
analysis applied to textile goods, Textile Res. J. 74, 1066–1072.
Picard, D. (2007). Tactual, visual, and cross-modal transfer of texture in 5- and 8-year-old chil-
dren, Perception 36, 722–736.
Picard, D., Dacremont, C., Valentin, D. and Giboreau, A. (2003). Perceptual dimensions of
tactile textures, Acta Psychol. 114, 165–184.
Piqueras-Fiszman, B., Laughlin, Z., Miodownik, M. and Spence, C. (2012). Tasting spoons:
assessing how the material of a spoon affects the taste of the food, Food Qual. Prefer. 24,
24–29. DOI:10.1016/j.foodqual.2011.08.005.
Piqueras-Fiszman, B. and Spence, C. (2012). The weight of the bottle as a possible extrinsic
cue with which to estimate the price (and quality) of the wine? Observed correlations, Food
Qual. Pref. 25, 41–45.
Piqueras-Fiszman, B. and Spence, C. (2015). Sensory expectations based on product-extrinsic
food cues: an interdisciplinary review of the empirical evidence and theoretical accounts,
Food Qual. Pref. 40, 165–179.
Podrebarac, S. K., Goodale, M. A. and Snow, J. C. (2014). Are visual texture-selective areas
recruited during haptic texture discrimination?, NeuroImage 94, 129–137.
Posner, M. I., Nissen, M. J. and Klein, R. M. (1976). Visual dominance: an information-
processing account of its origins and significance, Psychol. Rev. 83, 157–171.
Pursey, T. and Lomas, D. (2018). Tate sensorium: an experiment in multisensory immersive
design, Senses Soc. 13, 354–366.
Pye, E. (2007). The Power of Touch: Handling Objects in Museums and Heritage Contexts.Left
Coast Press, Walnut Creek, CA, USA.
Qiao, X., Wang, P., Li, Y. and Hu, Z. (2014). Study on a correlation model between the kansei
image and the texture harmony, Int. J. Signal Proc. Image Proc. Pattern Recognit. 7, 73–84.
Rabin, M. D. (1988). Experience facilitates olfactory quality discrimination, Percept. Psy-
chophys. 44, 532–540.
Reinoso Carvalho, F., Wang, Q. (J.), Van Ee, R., Persoone, D. and Spence, C. (2017). ‘Smooth
operator’: music modulates the perceived creaminess, sweetness, and bitterness of chocolate,
Appetite 108, 383–390.
Riegl, A. (1901). Die Spätrömische Kunst-Industrie Nach den Funden in Österreich-Ungarn.
Druck und Verlag der Kaiserlich-Königlichen Hof- und Staatsdruckerei, Vienna, Austra.
Roberts, J. R., Jones, R., Mansfield, N. J. and Rothberg, S. J. (2005). Evaluation of impact sound
on the ‘feel’ of a golf shot, J. Sound Vib. 287, 651–666.
Rock, I. and Harris, C. S. (1967). Vision and touch, Sci. Am. 216, 96–107.
Rock, I. and Victor, J. (1964). Vision and touch: an experimentally created conflict between the
two senses, Science 143, 594–596.
Saad, G. and Gill, T. (2000). Applications of evolutionary psychology in marketing, Psychol.
Mark. 17, 1005–1034.
Salisbury, K., Brock, D., Massie, T., Swarup, N. and Zilles, C. (1995). Haptic rendering: pro-
gramming touch interaction with virtual objects, in: Proceedings of the 1995 Symposium on
Interactive 3D Graphics (I3D’95). ACM, New York, New York, pp. 123–130.
Salisbury, K., Conti, F. and Barbagli, F. (2004). Haptic rendering: introductory concepts, IEEE
Comput. Graph. Appl. 24, 24–32.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 771
Sathian, K., Lacey, S., Stilla, R., Gibson, G. O., Deshpande, G., Hu, X., LaConte, S. and Glielmi,
C. (2011). Dual pathways for haptic and visual perception of spatial and texture information,
NeuroImage 57, 462–475.
Schifferstein, H. N. J. and Hekkert, P. (2008). Product Experience. Elsevier, London, UK.
Schifferstein, H. N. J. and Hekkert, P. (2011). Multisensory aesthetics in product design, in:
Art and the Senses, F. Bacci and D. Melcher (Eds), pp. 543–569. Oxford University Press,
Oxford, UK.
Schifferstein, H. N. J. and Michaut, A. M. K. (2002). Effects of appropriate and inappropriate
odors on product evaluations, Percept. Mot. Skills 95, 1199–1214.
Schifferstein, H. N. J. and Spence, C. (2008). Multisensory product experience, in: Product
Experience, H. N. J. Schifferstein and P. Hekkert (Eds), pp. 133–161. Elsevier, London, UK.
Schiffman, S. S. (1974). Physicochemical correlates of olfactory quality, Science 185, 112–117.
Schneider, I. K., Rutjens, B. T., Jostmann, N. B. and Lakens, D. (2011). Weighty matters: im-
portance literally feels heavy, Soc. Psychol. Personal. Sci. 2, 474–478.
Schütte, S., Nagamachi, M., Schütte, S. and Eklund, J. (2008). Affective meaning: the kansei
engineering approach, in: Product Experience, H. N. J. Schifferstein and P. Hekkert (Eds),
pp. 477–496. Elsevier, London, UK.
Senna, I., Maravita, A., Bolognini, N. and Parise, C. V. (2014). The marble-hand illusion, PLoS
ONE 9, e91688. DOI:10.1371/journal.pone.0091688.
Serafin, S., Fontana, F., Turchet, L. and Papetti, S. (2011). Auditory rendering and display of
interactive floor cues, in: Walking With the Senses – Perceptual Techniques for Walking in
Simulated Environments, F. Fontana and Y. Visell (Eds), pp. 123–152. Logos Verlag, Berlin,
Germany.
Sharan, L., Rosenholtz, R. and Adelson, E. H. (2014). Accuracy and speed of material catego-
rization in real-world images, J. Vis. 14, 12. DOI:10.1167/14.9.12.
Sheldon, R. and Arens, E. (1932). Consumer Engineering: a New Technique for Prosperity.
Harper and Brothers Publishers, New York, NY, USA.
Sonneveld, M. H. and Schifferstein, H. N. J. (2008). The tactual experience of objects, in: Prod-
uct Experience, H. N. J. Schifferstein and P. Hekkert (Eds), pp. 41–67. Elsevier, London,
UK.
Spence, C. (2002). The ICI Report on the Secret of the Senses. The Communication Group,
London, UK.
Spence, C. (2007). Making sense of touch: a multisensory approach to the perception of objects,
in: The Power of Touch: Handling Objects in Museums and Heritage Contexts, E. Pye (Ed.),
pp. 45–61. Left Coast Press, Walnut Creek, CA, USA.
Spence, C. (2009a). Measuring the impossible, in: Proceedings of the MINET Conference: Mea-
surement, Sensation and Cognition, pp. 53–61. National Physical Laboratories, Teddington,
UK.
Spence, C. (2011a). The multisensory perception of touch, in: Art and the Senses, F. Bacci and
D. Melcher (Eds), pp. 85–106. Oxford University Press, Oxford, UK.
Spence, C. (2011b). Crossmodal correspondences: a tutorial review, Atten. Percept. Psychophys.
73, 971–995.
Spence, C. (2011c). Crystal clear or gobbletigook?, World Fine Wine 33, 96–101.
Spence, C. (2015). Eating with our ears: assessing the importance of the sounds of consumption
to our perception and enjoyment of multisensory flavour experiences, Flavour 4, 3. DOI:10.
1186/2044-7248-4-3.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

772 C. Spence / Multisensory Research 33 (2020) 737–775
Spence, C. (2017). Gastrophysics: the New Science of Eating. Viking Penguin, London, UK.
Spence, C. (2019). Tactile/haptic aspects of multisensory packaging design, in: Multisensory
Packaging: Designing New Product Experiences, C. Velasco and C. Spence (Eds), pp. 127–
159. Palgrave MacMillan, Cham, Switzerland.
Spence, C. (in press). Temperature-based crossmodal correspondences: causes & consequences,
Multisens. Res. DOI:10.1163/22134808-20191494.
Spence, C. and Gallace, A. (2011). Multisensory design: reaching out to touch the consumer,
Psychol. Mark. 28, 267–308.
Spence, C. and Piqueras-Fiszman, B. (2011). Multisensory design: weight and multisensory
product perception, in: Proceedings of RightWeight2, G. Hollington (Ed.), pp. 8–18. Mate-
rials KTN, London, UK.
Spence, C. and Piqueras-Fiszman, B. (2014). The Perfect Meal: the Multisensory Science of
Food and Dining. Wiley-Blackwell, Oxford, UK.
Spence, C. and Wang, Q. (J.) (2015). Sensory expectations elicited by the sounds of opening
the packaging and pouring a beverage, Flavour 4, 35. DOI:10.1186/s13411-015-0044-y.
Spence, C. and Wang, Q. (J.) (2017). Assessing the impact of closure type on wine ratings and
mood, Beverages 3, 52. DOI:10.3390/beverages3040052.
Spence, C. and Zampini, M. (2006). Auditory contributions to multisensory product perception,
Acta Acust. United Acust. 92, 1009–1025.
Spence, C. and Zampini, M. (2007). Affective design: modulating the pleasantness and force-
fulness of aerosol sprays by manipulating aerosol spraying sounds, CoDesign 3(Suppl. 1),
107–121.
Stanton, T. R. and Spence, C. (2020). The influence of auditory cues on bodily and movement
perception, Front. Psychol. 10, 3001. DOI:10.3389/fpsyg.2019.03001.
Stuckey, B. (2012). Taste What You’re Missing: the Passionate Eater’s Guide to Why Good
Food Tastes Good. Free Press, London, UK.
Sun, H.-C., Welchman, A. E., Chang, D. H. F. and Di Luca, M. (2016). Look but don’t touch:
visual cues to surface structure drive somatosensory cortex, NeuroImage 128, 353–361.
Sundar, A. and Noseworthy, T. J. (2016a). When sensory marketing works and when it backfires.
Harvard Business Review, May 19th. Retrieved from https://hbr.org/2016/05/when-sensory-
marketing-works-and-when-it-backfires?referral=03758&cm_vc=rr_item_page.top_right.
Sundar, A. and Noseworthy, T. J. (2016b). Too exciting to fail, too sincere to succeed: the effects
of brand personality on sensory disconfirmation, J. Consum. Res. 43, 44–67.
Susini, P., McAdams, S., Winsberg, S., Perry, I., Vieillard, S. and Rodet, X. (2004). Character-
izing the sound quality of air-conditioning noise, Appl. Acoust. 65, 763–790.
Suzuki, Y., Gyoba, J. and Sakamoto, S. (2008). Selective effects of auditory stimuli on tactile
roughness perception, Brain Res. 1242, 87–94.
Suzuki, Y., Suzuki, M. and Gyoba, J. (2006). Effects of auditory feedback on tactile roughness
perception, Tohoku Psychol. Folia 65, 45–56.
Tani, Y., Nagai, T., Koida, K., Kitazaki, M. and Nakauchi, S. (2014). Experts and novices use
the same factors—but differently—to evaluate pearl quality, PLoS One 9, e86400. DOI:10.
1371/journal.pone.0086400.
Teli, M. D. (2015). Softening finishes for textiles and clothing, in: Functional Finishes for
Textiles: Improving Comfort, Performance and Protection, Series in Textiles, Number 156,
P. Roshan (Ed.), pp. 123–152. Woodhead Publishing, Cambridge, UK.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

C. Spence / Multisensory Research 33 (2020) 737–775 773
Tonetto, L., Klanovicz, C. P. and Spence, C. (2014). Modifying action sounds influences peo-
ple’s emotional responses and bodily sensations, i-Perception 5, 153–163.
Turchet, L., Serafin, S., Dimitrov, S. and Nordahl, R. (2010). Conflicting audio-haptic feed-
back in physically based simulation of walking sounds, in: Haptic and Audio Interaction
Design. HAID 2010,Lecture Notes in Computer Science, Vol. 6306, R. Nordahl, S. Serafin,
F. Fontana and S. Brewster (Eds), pp. 47–52. Springer, Berlin, Germany.
Ueda, J., Spence, C. and Okajima, K. (submitted). Effects of luminance distribution on food
appearance and taste perception, Sci. Rep.
Van Lente, R. and Herman, S. J. (2001). The smell of success — exploiting the leather aroma,
SAE Technical Paper 2001-01-0047. DOI:10.4271/2001-01-0047.
Velasco, C. and Spence, C. (2019). Multisensory Packaging: Designing New Product Experi-
ences. Palgrave MacMillan, Cham, Switzerland.
Velasco, C., Jones, R., King, S. and Spence, C. (2013a). ‘Hot or cold?’ On the informative
value of auditory cues in the perception of the temperature of a beverage, in: (((ABA)))
Audio Branding Academy Yearbook 2012/2013, K. Bronner, R. Hirt and C. Ringe (Eds),
pp. 175–187. Nomos, Baden-Baden, Germany.
Velasco, C., Jones, R., King, S. and Spence, C. (2013b). The sound of temperature: what infor-
mation do pouring sounds convey concerning the temperature of a beverage, J. Sens. Stud.
28, 335–345.
Velasco, C., Michel, C., Youssef, J., Gamez, X., Cheok, A. D. and Spence, C. (2016). Colour–
taste correspondences: designing food experiences to meet expectations or to surprise, Int.
J. Food Des. 1, 83–102.
Vickers, G. and Spence, C. (2007). Get set for the sensory side of the century. Contact: Royal
Mail’s Magazine for Marketers, November, 11–14.
Walker-Andrews, A. (1994). Taxonomy for intermodal relations, in: The Development of In-
tersensory Perception: Comparative Perspectives, D. J. Lewkowicz and R. Lickliter (Eds),
pp. 39–56. Lawrence Erlbaum, Hillsdale, NJ, USA.
Walker, P., Francis, B. J. and Walker, L. (2010). The brightness-weight illusion: darker objects
look heavier but feel lighter, Exp. Psychol. 57, 462–469.
Walker, P., Scallon, G. and Francis, B. (2017). Cross-sensory correspondences: heaviness is
dark and low-pitched, Perception 46, 772–792.
Walker, P., Scallon, G. and Francis, B. J. (in press). Heaviness–brightness correspondence
and stimulus-response compatibility, Atten. Percept. Psychophys. DOI:10.3758/s13414-
019-01963-6.
Wallmark, Z. (2019). Semantic crosstalk in timbre perception, Music Sci. 2, 1–18.
Wang, Q. (J.) and Spence, C. (2017). The role of pitch and tempo in sound–temperature cross-
modal correspondences, Multisens. Res. 30, 307–320.
Wang, Q. J. and Spence, C. (2019). Sonic packaging: how packaging sounds influence multi-
sensory product evaluation, in: Multisensory Packaging, C. Velasco and C. Spence (Eds),
pp. 103–125. Palgrave MacMillan, Cham, Switzerland.
Warren, D. H. and Rossano, M. J. (1991). Intermodality relations: vision and touch, in: The
Psychology of Touch, M. A. Heller and W. Schiff (Eds), pp. 119–137. Lawrence Erlbaum,
Hillsdale, NJ, USA.
Wastiels, L., Schifferstein, H. N. J., Heylighen, A. and Wouters, I. (2012). Red or rough, what
makes materials warmer?, Mater. Des. 42, 441–449.
Downloaded from Brill.com02/11/2023 01:47:54AM
via free access

774 C. Spence / Multisensory Research 33 (2020) 737–775
Wastiels, L., Schifferstein, H. N. J., Wouters, I. and Heylighen, A. (2013). Touching materials
visually: about the dominance of vision in building material assessment, Int. J. Des. 7, 31–
41.
Werner, H. and von Schiller, P. (1932). Untersuchungen über Empfindung und Empfinden: 5.
Rauhigkeit als intermodale Erscheinung, Z. Psychol. 127, 265–289.
Whitaker, T. A., Simões-Franklin, C. and Newell, F. N. (2008). The natural truth: the contri-
bution of vision and touch in the categorisation of ‘naturalness’, in: Haptics: Perception,
Devices and Scenarios, Eurohaptics 2008,Lecture Notes in Computer Science, Vol. 5024,
M. Ferre (Ed.). Springer, Berlin, Germany.
Whitsel, B. L., Favorov, O., Tommerdahl, M., Diamond, M., Juliano, S. L. and Kelly, D. G.
(1989). Dynamic processes governing the somatosensory cortical response to natural stimu-
lation, in: Sensory Processing in the Mammalian Brain, J. S. Lund (Ed.), pp. 84–116. Oxford
University Press, New York, NY, USA.
Wijntjes, M. W. A., Xiao, B. and Volcic, R. (2019). Visual communication of how fabrics feel,
J. Vis. 19, 4. DOI:10.1167/19.2.4.
Wildes, R. P. and Richards, W. A. (1988). Recovering material properties from sound, in: Nat-
ural Computation, W. A. Richards (Ed.), pp. 356–363. MIT Press, Cambridge, MA, USA.
Wolkomir, R. (1996). Decibel by decibel, reducing the din to a very dull roar, Smithsonian Mag.
26, 56–64.
Workman, J. E. (2010). Fashion consumer groups, gender, and need for touch, Clothing Text.
Res. J. 28, 126–139.
Wright, R. H. (1966). Why is an odour?, Nature 209, 551–554.
Xiao, B., Bi, W., Jia, X., Wei, H. and Adelson, E. H. (2016). Can you see what you feel? Color
and folding properties affect visual–tactile material discrimination of fabrics, J. </