Converging coolness and investigating its relation to user experience

Abstract

Recently a number of studies appeared that operationalised coolness and explored its relation to digital products. Literature suggests that perceived coolness is another factor of user experience, and this adds to an existing explosion of dimensions related to aesthetics, hedonic quality, pragmatic quality, attractiveness, etc. A critical challenge highlighted in prior research is to study the relationships among those factors and so far, no studies have empirically examined the relationship between coolness and other established user experience factors. In this paper, we address this challenge by presenting two studies one that focuses on factors from two cool questionnaires, and one that compares them against existing User eXperience (UX) factors. Our findings show that factors from the two cool questionnaires converge and they also converge to existing, established UX factors. Thus, 11 distinct cool and UX factors converge into 5 for the case of mobile devices. Our findings are important for researchers, as we demonstrate through a validated model that coolness is part of UX research, as well as for practitioners, by developing a questionnaire that can reliably measure both perceived inner and outer coolness as well as the overall coolness judgement based on 5 factors and 21 items.

Full text

Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=tbit20
Download by: [Aalborg University Library] Date: 07 October 2016, At: 00:11
Behaviour & Information Technology
ISSN: 0144-929X (Print) 1362-3001 (Online) Journal homepage: http://www.tandfonline.com/loi/tbit20
Converging coolness and investigating its relation
to user experience
Dimitrios Raptis, Anders Bruun, Jesper Kjeldskov & Mikael B. Skov
To cite this article: Dimitrios Raptis, Anders Bruun, Jesper Kjeldskov & Mikael B. Skov (2016):
Converging coolness and investigating its relation to user experience, Behaviour & Information
Technology, DOI: 10.1080/0144929X.2016.1232753
To link to this article: http://dx.doi.org/10.1080/0144929X.2016.1232753
Published online: 06 Oct 2016.
Submit your article to this journal
View related articles
View Crossmark data

Converging coolness and investigating its relation to user experience
Dimitrios Raptis, Anders Bruun, Jesper Kjeldskov and Mikael B. Skov
Department of Computer Science, Aalborg University, Aalborg Oest, Denmark
ABSTRACT
Recently a number of studies appeared that operationalised coolness and explored its relation to
digital products. Literature suggests that perceived coolness is another factor of user experience,
and this adds to an existing explosion of dimensions related to aesthetics, hedonic quality,
pragmatic quality, attractiveness, etc. A critical challenge highlighted in prior research is to study
the relationships among those factors and so far, no studies have empirically examined the
relationship between coolness and other established user experience factors. In this paper, we
address this challenge by presenting two studies one that focuses on factors from two cool
questionnaires, and one that compares them against existing User eXperience (UX) factors. Our
findings show that factors from the two cool questionnaires converge and they also converge to
existing, established UX factors. Thus, 11 distinct cool and UX factors converge into 5 for the
case of mobile devices. Our findings are important for researchers, as we demonstrate through a
validated model that coolness is part of UX research, as well as for practitioners, by developing a
questionnaire that can reliably measure both perceived inner and outer coolness as well as the
overall coolness judgement based on 5 factors and 21 items.
ARTICLE HISTORY
Received 24 February 2016
Accepted 30 August 2016
KEYWORDS
Coolness; inner cool; outer
cool; user experience;
dimensionality explosion;
questionnaires
1. Introduction
For more than a decade, User eXperience (UX) has been
applied as a broad notion to describe experienced qual-
ities of interactive products and UX research focuses
on exploring the experiential, affective, meaningful,
and valuable aspects of product use (Vermeeren et al.
2010). UX goes beyond the instrumental emphasis of
usability (Bargas-Avila and Hornbæk 2011) and
although the satisfaction part of usability is considered
as a relevant dimension for UX (Law et al. 2009), UX
qualities are not limited to that. In this paper we focus
on factors that, according to the literature, contribute
to coolness and we study, firstly, how they converge
and shape the cool perception and, secondly, how they
relate to other subjective, measurable UX factors, such
as affect, enjoyment, fun, aesthetics, appeal, attractive-
ness, hedonic quality, engagement, flow, enchantment,
and frustration (Bargas-Avila and Hornbæk 2011).
The challenge for UX research is that the sheer volume
of factors has reached such a large number where it is
critical to start discussing the extent these are converging.
Bargas-Avila and Hornbæk (2011) apply the term
‘dimensionality explosion’to denote this phenomenon
within UX research. There is a need for the UX research
community to study this explosion, for example, the
relation between hedonic quality and attractiveness. Are
we referring to the same or a similar factor with two
different names? In every context? For every product?
Bargas-Avila and Hornbæk (2011) suggest that dimen-
sionality explosion occurs, firstly, because many of
these factors are not established as they have not been
tested for their reliability and validity (many researchers
use self-made items without providing them), and, sec-
ondly, because several proposed factors are not posi-
tioned in relation to the rest (the main problem of this
dimensionality explosion is that the relation to established
constructs is rarely made clear). For example, no one to
our knowledge has compared hedonic quality and attrac-
tiveness and produced specific results on if (or how) they
converge into one factor. Besides this research challenge,
this situation also creates significant problems for prac-
titioners too, as there is no agreement on which question-
naires to use, and under which conditions.
A recent example of an emerging new perceived UX fac-
tor is coolness. In the past five years, the human-computer
interaction (HCI) community has increased its focus on
determining coolness of digital products, and ‘designing
for cool’is becoming an essential criterion when develop-
ing new applications, interfaces, and devices (Sundar
et al. 2014). The main driving force behind this research
effort was the fact that the term coolness has often been
used by people to positively describe their experiences
with various products such as cars, home appliances,
mobile phones, etc. (Raptis, Kjeldskov, and Skov 2013).
© 2016 Informa UK Limited, trading as Taylor & Francis Group
CONTACT Anders Bruun bruun@cs.aau.dk
BEHAVIOUR & INFORMATION TECHNOLOGY, 2016
http://dx.doi.org/10.1080/0144929X.2016.1232753

Until recently, relatively few studies investigated cool-
ness within HCI. Holtzblatt (2011) discussed the concept
of cool and emphasised that coolness contributes to our
personal feelings of accomplishment, connection with
others, identity, and delightful experiences. Read, Hor-
ton, and Fitton (2012) developed a framework on
‘Being Cool’,by‘Doing Cool Things’and by ‘Having
Cool Stuff’. The aim of that framework is to facilitate
the design of cool products for teenagers based on factors
such as being rebellious, antisocial, retro, authentic, rich,
and innovative. Culén and Gasparini (2012) argued that
product coolness is related to fun, mastery, adding value,
useful, successful, self-presentation, and innovation. A
more comprehensive overview on coolness can be
found in Raptis, Kjeldskov, and Skov (2013).
In the above-mentioned studies, the identified coolness
characteristics were derived on the basis of literature
reviews, which have made considerable contributions in
shaping our initial understanding of coolness. McCrick-
ard, Barksdale, and Doswell (2012) moved a step further
and explicated the need for a ‘cool engineering’approach
to support designing for coolness. The aim of such an
approach would be to understand how target users per-
ceive coolness of products in various contexts of use. In
order to define a cool engineering approach, firstly, there
is a need to produce reliable tools and techniques for
measuring coolness. Towards this end, two similar studies
recently focused on breaking down the concept of cool
into smaller entities and produced questionnaires that
reliably measure perceived coolness (Bruun et al. 2016;
Sundar et al. 2014). Sundar et al. (2014) produced a ques-
tionnaire with 15 items that measures coolness through
factors related to subculture, attractiveness, and orig-
inality. The COOL Questionnaire proposed by Bruun
et al. (2016) consists of 16 items and it builds on a distinc-
tion between inner and outer coolness. Bruun et al. (2016)
measured the perceived inner coolness of interactive pro-
ducts through factors related to usability, desirability, and
rebelliousness and they also suggest that perceived outer
coolness is related to attractiveness and aesthetics.
Consequently, at this moment there are two question-
naires that measure coolness through differently labelled
factors. Since some of these factors are seemingly com-
parable (e.g. subculture and rebelliousness) and by taking
into consideration the problem of the dimensionality
explosion, there is a need both to examine the extent
of convergence between proposed coolness factors, and
to examine how coolness and existing UX factors con-
verge. Thus, the aim of our paper is twofold:
(1) To make a systematic comparison of the underlying
factors of the two recently proposed coolness
questionnaires;
(2) To compare these coolness factors against estab-
lished UX factors.
In this paper we report two studies, one for each aim.
In the following, we provide an overview of existing
questionnaires for measuring perceived coolness and
other UX-related factors (Sections 2 and 3). In Section
4, we outline a set of hypotheses on how we expect the
coolness factors to converge and how we expect them
to converge with other UX factors. Section 5 describes
the method, and then Sections 6 and 7 highlight our
results. In Section 8, we discuss our findings against
our research hypotheses as well as their implications
for research and practice. Finally, in Section 9 we con-
clude our paper by highlighting the most important
findings.
2. Established questionnaires for measuring
coolness
In this section, we present in detail two recently pub-
lished questionnaires for measuring perceived coolness
and their underlying factors.
2.1. ‘Capturing cool’
In 2014, the paper ‘Capturing cool: Measures for asses-
sing perceived coolness of technological products’was
published by Sundar et al. The aim of that study was
to produce a questionnaire for measuring coolness.
Through the literature, Sundar et al. (2014) identified a
set of four factors that characterised coolness. The first
factor is based on the work by Kerner, Pressman, and
Essex (2007) and Levy (2006) and relates to the unique-
ness of a product over competing products. A second
factor is related to product attractiveness, which has its
roots within aesthetics. Its theoretical underpinning
was based on the work of Levy (2006) and Tractinsky
(1997). According to Sundar et al., attractiveness encom-
passes the externally visible aesthetic appeal, but is also
related to the social acceptability of a given style, for
example, whether or not a product makes the owner
look good in relation to others. Their third factor deals
with the subcultural aspects of products. Based on the
work by Dar-Nimrod et al. (2012) and Horton et al.
(2012), the authors suggest that subculture includes an
element of rebelliousness, for example, that a product
appeals to a minority (and not the mainstream consumer
group) by being ‘edgy’. According to Sundar et al. (2014),
subculture also deals with the utility of a product for a
particular group of persons. So, a product can be con-
sidered cool if it is useful for a specific group, while indi-
cating one’s affiliation to this particular group. Their
2D. RAPTIS ET AL.

fourth and final factor relates to genuineness. Based on
Conan (2008), Kerner, Pressman, and Essex (2007),
and Levy (2006), this factor is about authenticity and
the sincere nature of a product. Thus, a product, of
which the underlying intentions of its designer are to
really improve the lives of its users, is perceived as
genuine.
Sundar et al. applied these 4 factors as an offset to cre-
ate 35 evaluative statements of coolness, for example,
‘The designers of this product primarily want to create
better products’(related to genuineness). An exploratory
study with 315 participants was conducted and key fac-
tors within the 35 question items were extracted. This
was followed by an additional 2 studies based on 1150
respondents assessing the coolness of various products
such as USB drives, Nintendo Wii, Prezi, Warcraft, etc.
The statistical techniques of exploratory and confirma-
tory factor analyses (CFAs) were applied throughout
the studies, and Figure 1 shows their final three-factor
structure of perceived coolness, which can be measured
through the factors of subculture, attractiveness, and
originality and 15 items. Each item is represented on a
nine-item Likert scale (1 = Strongly disagree, 9 =
Strongly agree). Furthermore, they demonstrated that
these three factors do contribute to the overall coolness
judgement for a product by comparing them to a set of
nine items that measured overall coolness (Table 1).
2.2. ‘The COOL questionnaire’
Bruun et al. (2016) similarly focused on breaking down
the concept of cool into smaller entities and used them
as building blocks to produce the ‘COOL Questionnaire’
1
(Bruun et al. 2016). The process is similar to what Sundar
et al. (2014) used to create their ‘Capturing Cool’ques-
tionnaire, where factors are derived on a theoretical
basis followed by a set of statistical studies.
Bruun et al.’s(2016) study differs from the Sundar
et al. (2014) study as they propose that coolness is
decomposed to inner cool and outer cool. They base
this distinction on a literature review presented in Raptis,
Kjeldskov, and Skov (2013). According to Nancarrow,
Nancarrow, and Page (2001) and MacAdams (2001),
inner coolness deals with the personality of someone,
that is, how others perceive intra-person characteristics.
As an example, a person can be perceived as cool if he or
she keeps his or her calm under pressure. Inner cool in
relation to products refers to the perceived personality
traits, which are assigned to products by users, for
example, a product can be considered as cute or tough
(Janlert and Stolterman 1997; Jordan 1997). Outer cool
relates to how something or someone is presented
through a certain style in physical appearance (Gioia
2009). For products, this is a matter of aesthetic design,
for example, physical shape, materials, colours, and so
on.
The questionnaire presented in Bruun et al. (2016)
measures the perceived inner coolness of products, but
not outer coolness. Authors suggest that outer cool is
directly related to aesthetic attributes and thus it can
be measured by existing UX factors, for example, by
using questionnaires that measure attractiveness or aes-
thetics such as the one proposed in Lavie and Tractinsky
(2004). In a similar matter as the Sundar et al. (2014)
study, Bruun et al. (2016) start the process of creating
their questionnaire by identifying relevant characteristics
that contribute to inner coolness. Informed by a litera-
ture review (Raptis, Kjeldskov, and Skov 2013), they
identified 11 characteristics that contribute to inner cool-
ness, namely, being rebellious and antisocial, embracing
authenticity and innovation, seeking exclusivity, pleasure
and personal development, being/appearing in control,
making hard things appear easy, being detached/
emotionally neutral, and being strongly tight to a
group. The majority of the 11 characteristics emerged
from the work of Pountain and Robbins (2000), MacA-
dams (2001), and Nancarrow, Nancarrow, and Page
(2001). For more details, we refer to Raptis, Kjeldskov,
and Skov (2013).
Bruun et al. derived their questionnaire using an
initial pool of 143 items related to the eleven inner
Figure 1. Three-factor structure of cool (Sundar, Tamul, and Wu
2014).
Table 1. Factors and items from the ‘Capturing cool’
questionnaire (Sundar, Tamul, and Wu 2014).
Subculture Attractiveness Originality
This device makes people who
use it different from other
people
This device is stylish This device is original
If I used this device, it would
make me stand apart from
others
This device is hip This device is unique
This device helps people who
use it to stand apart from
the crowd
This device is sexy This product is novel
People who use this product
are unique
This device is hot This device is out of
the ordinary
People who use this device
would be considered
leaders rather than
followers
This device is on
the cutting edge
This product stands
apart from similar
products
BEHAVIOUR & INFORMATION TECHNOLOGY 3

cool characteristics. Through an iterative process with a
total of 2236 respondents and by repeatedly applying the
statistical techniques of exploratory and confirmatory
factor analyses, they ended up having a questionnaire
with 16 items (Table 2) distributed to three factors of
desirability, rebelliousness, and perceived usability
(Figure 2). All items are measured on a 7-point Likert
scale (1 = Strongly disagree, 7 = Strongly agree).
Additionally and similarly to Sundar et al. (2014),
Bruun et al. also demostrated that these three factors
shape the overall coolness judgement, which was
measured through the item ‘This device is cool’.
3. Questionnaires for measuring established
UX factors
As mentioned in the introduction of the paper, Bargas-
Avila and Hornbæk (2011) point towards the existence
of a ‘dimensionality explosion’in relation to measuring
UX. A critical question here is whether coolness further
fuels this explosion and if its factors, to some extent,
overlap with existing factors. In order to answer this
question, a set of established UX factors are presented
in the following paragraphs.
Many widely considered UX factors concern the aes-
thetic appeal of interaction designs (Bargas-Avila and
Hornbæk 2011). Lavie and Tractinsky (2004) proposed
a questionnaire for assessing the level of website aes-
thetics. Their questionnaire has since then been applied
to evaluate UX of various products, such as mobile
phones (Sonderegger et al. 2012). The questionnaire is
based on the two factors of classic and expressive
aesthetics. Items of the classic aesthetics factor consist
of a set of adjectives such as ‘Pleasant’,‘Clean’, and ‘Sym-
metric’. Lavie and Tractinsky (2004) state that this factor
deals with traditional notions of aesthetics. The expres-
sive aesthetics factor represents qualities that go beyond
the classical design principles and includes items such as
‘Creative’,‘Fascinating’, and ‘Sophisticated’.Table 3
shows all the questionnaire items, which are rated on a
7-point Likert scale (1 = Strongly disagree, 7 = Strongly
agree).
Attrakdiff (Hassenzahl, Burmester, and Koller 2003)
is also a widely recognised questionnaire for measuring
UX (Bargas-Avila and Hornbæk 2011). Like the aes-
thetics questionnaire, it was created with a focus on web-
sites, but it has also been successfully applied to assess
the UX of various types of products, for example, cultu-
rally adaptive applications (Reinecke and Bernstein
2011). The shortened version of this questionnaire
(Attrakdiff2) is based on a two-factor structure concern-
ing hedonic and pragmatic qualities and two evaluative
constructs (Van Schaik, Hassenzahl, and Ling 2012).
The hedonic quality factor deals with the overall appeal
of a product and includes items related to aesthetics
(e.g. ‘I judge the product to be stylish’) as well as items
about excitement (e.g. ‘I judge the product to be captivat-
ing’). The pragmatic quality factor revolves around utili-
tarian and usability aspects with underlying items such
as ‘I judge the product to be confusing/structured’or ‘I
judge the product to be impractical/practical’. All items
(Table 4) are assessed on a 7-point scale (e.g. 1 = dull,
7 = captivating).
Finally, attractiveness is also considered as an estab-
lished UX factor and since it was identified as a core fac-
tor in the Sundar study, we chose to include a reliable
Table 2. Factors and items from the ‘COOL questionnaire’(Bruun
et al. 2016).
Desirability Rebelliousness Perceived usability
This device can make me
better
This device moves
against the current
This device is easy
to operate
This device is meant for
people like me
This device is
unconventional
This device is easy
to use
This device can make me
happy
This device is different This device is easy
to learn
This device can make me
look good
This device is outside the
ordinary
This device is
simple to use
This device totally connects
with me
This device is rebellious This device is
effortless to use
This device can make me
look in control of things
Figure 2. Three-factor structure of inner cool (Bruun et al. 2016).
Table 3. Factors and items from the aesthetics questionnaire
(Lavie and Tractinsky 2004).
Classic aesthetics Expressive aesthetics
This device has: This device has:
Aesthetic design Creative design
Pleasant design Fascinating design
Clear design Use of special effects
Clean design Original design
Symmetric design Sophisticated design
Table 4. Factors and items from Attrakdiff2 questionnaire (Van
Schaik, Hassenzahl, and Ling 2012).
Hedonic quality Pragmatic quality Evaluative constructs
I judge the device to
be:
I judge the device to be: I judge the device overall to
be:
Dull-Captivating Confusing-Structured Bad-Good
Tacky-Stylish Impractical-Practical Ugly-Beautiful
Cheap-Premium Unpredictable-
Predictable
Unimaginative-
Creative
Complicated-Simple
4D. RAPTIS ET AL.

questionnaire that measures it. Quinn and Tran (2010)
developed a five 7-point scale to measure the attractive-
ness of a product and they used it to assess the attractive-
ness of mobile phones. The underlying items (Table 5)
not only deal with aesthetic notions similar to those
suggested in Lavie and Tractinsky (2004), but they also
relate to the hedonic factor of Van Schaik, Hassenzahl,
and Ling (2012). In order to differentiate between the
attractiveness factor of Sundar et al. (2014) and the
attractiveness factor of Quinn and Tran (2010), we will
refer to the first as attractiveness(cool) and the latter as
attractiveness(UX).
4. Research hypotheses
As pointed out by Bargas-Avila and Hornbæk (2011),
when a new UX factor is proposed, the tools that
measure it should be tested for their reliability and val-
idity, and the factor should also be compared against
other established UX factors. In this paper, we report
two studies that follow this suggestion. Firstly, we com-
pared the two existing cool questionnaires alone (study
1), and then we compared them against the established
UX factors presented in the previous section (study 2).
In the following, we discuss our hypotheses in relation
to how cool factors and UX factors converge.
4.1. Study 1: Converging factors of coolness
At a first glance, the factors of attractiveness (Sundar
et al. 2014), desirability, and usability (Bruun et al.
2016) seem different; that is, they represent different
aspects of coolness. Based on their items, attractiveness
deals with aesthetic appeal (e.g. ‘This device is stylish’),
while desirability relates to personal desires (e.g. ‘This
device can make me happy’). Perceived usability is differ-
ent from these as it concerns perceived learnability, uti-
lity, and operability of the device (e.g. ‘This device is
effortless to use’).
However, there is some overlap between the question-
naires. In particular, we find the originality (Sundar et al.
2014) and rebelliousness (Bruun et al. 2016) factors to be
similar as they both deal with unconventional and novel
aspects of a product, for example, ‘This device is uncon-
ventional’vs. ‘This device is out of the ordinary’. The
subculture factor from Sundar et al. relates more to the
people using a product than the product itself. Yet, the
topic of being different and unique is essential (e.g.
‘This device makes people who use it different from
other people’).
Figures 1 and 2present the two existing models for
evaluating coolness of products, each with three factors.
Based on the seemingly comparable factors of orig-
inality/subculture and rebelliousness, we hypothesise
that a four-factor structure would emerge when combin-
ing the items from the two questionnaires (see Figure 3).
Thus, we hypothesise that the combination of items from
Sundar et al. (2014) and Bruun et al. (2016) would lead to
the following:
H1. Coolness can be measured through the four factors
of attractiveness, desirability, perceived usability, and
originality/subculture/rebelliousness.
4.2. Study 2: Converging factors of coolness and
established UX factors
We will start this section with our hypotheses on over-
lapping factors among established UX factors and we
will continue with their relation to the cool factors.
From Lavie and Tractinsky’s classic and expressive fac-
tors, there are items such as ‘Pleasant design’,or‘Fasci-
nating design’, which seem to overlap with items from
Van Schaik et al.’s hedonic factor, for example, ‘I judge
the product to be captivating’. These in turn are similar
to the items presented in Quinn and Tran’s attractive-
ness(UX) factor, for example, ‘I judge the product to
be interesting’. Thus, the factors of classic aesthetics,
Table 5. Items from the attractiveness questionnaire (Quinn and
Tran 2010).
Attractiveness (UX)
I judge the device to be:
Attractive-Unattractive
Beautiful-Ugly
Eye catching-Plain
Interesting-Boring
I like the way this phone looks
Figure 3. Hypothesised converging factors of cool when combining Sundar, Tamul, and Wu (2014) and Bruun et al. (2016).
BEHAVIOUR & INFORMATION TECHNOLOGY 5

expressive aesthetics, hedonic quality, and attractiveness
(UX) all deal with observable aesthetic characteristics.
This is in line with Hassenzahl and Monk (2010) and
Diefenbach, Kolb, and Hassenzahl (2014), who argue
that hedonic quality is similar to expressive aesthetics
in specific contexts. The pragmatic factor in Van Schaik,
Hassenzahl, and Ling (2012) seems to stand apart with
items related to the perceived usability of a product,
for example, ‘I judge this product to be complicated’.
Compared to the coolness questionnaires, we do see
similar items to those posed in the established UX ques-
tionnaires. The attractiveness(cool) factor suggested in
Sundar et al. (2014) deals with outer appearance and
its items seem comparable to those from classic/expres-
sive aesthetics/hedonic quality/attractiveness(UX) pre-
sented above. Also, the pragmatic quality factor of
AttrakDiff2 relates to the instrumental aspects of a pro-
duct, which is similar to the perceived usability factor
presented in Bruun et al.’s(2016) coolness questionnaire.
As an example, consider the item ‘I judge the product to
be simple’(Van Schaik, Hassenzahl, and Ling 2012) ver-
sus ‘This device is simple to use’(Bruun et al. 2016). The
hypothesised structure of coolness in Figure 3 also
suggests factors of originality/subculture/rebelliousness
and desirability. Respectively, these factors deal with
unconventional notions of a product and personal desire,
and, thus, seem to be independent from other established
UX factors.
On the basis of this discussion, we hypothesise that a
four-factor structure will emerge when comparing the
suggested factors of coolness (Bruun et al. 2016; Sundar
et al. 2014) and the established UX factors (Lavie and
Tractinsky 2004; Quinn and Tran 2010; Van Schaik,
Hassenzahl, and Ling 2012). This hypothesised four-fac-
tor structure is shown in Figure 4, along with two-way
arrows indicating suggested correlations between factors.
Thus, our second hypothesis (H2) relates coolness to
established UX factors:
H2a. The coolness factor of attractiveness(cool) con-
verges on established UX factors of classic/expressive
aesthetics/hedonic quality/attractiveness(UX).
H2b. The coolness factor of perceived usability con-
verges on the established UX factor of pragmatic quality.
H2c. The coolness factors of originality/subculture/
rebelliousness and desirability do not converge on any
of the established UX factors.
5. Method
In order to test our research hypotheses, we applied the
statistical techniques of exploratory factor analysis (EFA)
and CFA using SPSS v. 23 and AMOS v. 22, respectively.
EFA is based on an iterative process where items are
removed from an initial pool of items, based on how
much they contribute to measuring a particular factor.
We conducted two EFA studies, where we used the Bar-
tlett Test of Sphericity to test the homogeneity of var-
iances, the Kaiser–Meyer–Olkin Measure (KMO) to
test sampling adequacy, and Principal Axes Factoring
as an extraction method using an oblique rotation (as
recommended in the literature, e.g. Bulmer (1979) and
Field (2009)). The number of extracted factors was deter-
mined through a Scree test and through parallel analysis
(using Monte Carlo principal components analysis
(PCA), Watkins). In the two EFA studies, we removed
items by applying two criteria: low communalities (<.5)
and low factor loadings (<.65).
CFA is, as the name implies, of confirmatory nature
and it is used to validate the factor structure that
emerged though an EFA. In CFA there are not only
item loadings on factors, but also covariances between
factors, denoting how variances between any two pairs
of factors are correlated. The goodness of a factor
model is determined by a range of fit indices, which col-
lectively indicate whether or not the factor structure is
appropriate and reliable (Schreiber et al. 2006). In the
following sections, we present the used indices for each
CFA.
We conducted two CFAs to test our hypotheses based
on Structural Equation Modelling (SEM) with Maximum
Likelihood Estimation. When conducting SEM, it is
necessary to conduct a pre-analysis to examine whether
Figure 4. Hypothesised converging factors when combining coolness questionnaires (Bruun et al. 2016; Sundar, Tamul, and Wu 2014)
and established UX questionnaires (Lavie and Tractinsky 2004; Quinn and Tran 2010; Van Schaik, Hassenzahl, and Ling 2012). Attrac-
tiveness(UX) refers to the factor from Quinn and Tran (2010) and attractiveness(cool) refers to Sundar, Tamul, and Wu (2014).
6D. RAPTIS ET AL.

SEM assumptions are met in the data sample. These
assumptions are related to missing data, normality, line-
arity, and multicollinearity (Schreiber et al. 2006). We
had no missing data and the CFA datasets had univariate
normality with skewness values between −1 and .43 and
kurtosis values between −1.4 and .6. These are within
acceptable thresholds to assume that data are normally
distributed (Tabachnick and Fidell 2013). Due to the
strong factor loadings (>.6) identified during all EFAs,
we also assume linearity between latent and manifest
variables. The level of multicollinearity was also accepta-
ble according to Kutner, Nachtsheim, and Neter (2004)
with Variance Inflation Factor levels between 1.5 and
4.5. Therefore, in both CFA studies all the necessary
assumptions were met.
In our studies, we included a large number of partici-
pants to be able to do a statistical analysis. We recruited
these participants from Amazon Mechanical Turk
(MTurk). MTurk participants have been used success-
fully in other studies within HCI and have been shown
to provide valuable results (e.g. Boujarwah, Abowd,
and Arriaga 2012; Heer and Bostock 2010; Heimerl
et al. 2012). We limited our selection to people living
in the USA to avoid language barriers and to follow
the recommendations of Ross et al. (2010) and Huff
and Tingley (2015). Ross et al. (2010) conducted a profil-
ing study of MTurk workers and collected data for their
gender, age, income, and level of education. Their find-
ings show that the sample of US MTurk workers is
balanced in relation to income and gender, while there
are slightly more workers of younger age. Their edu-
cation level is similar to that of the whole US population
(OECD 2016). Huff and Tingley (2015) compared a large
sample of US MTurk workers to a nationally representa-
tive sample. They focused on age, gender, race, ideology,
occupation, and the areas participants live. The MTurk
sample was identical to the representative one in relation
to ideology, occupation, and the areas participants live,
and slightly imbalanced in relation to age, gender, and
race. As an additional quality measure, we only recruited
MTurk workers with 95% approval ratings, as rec-
ommended by Ross et al. (2010).
The need for a large pool of participants was satisfied
through MTurk also informed our experimental set-up.
Since it is not possible to interact physically with
MTurk workers and show them physical artefacts, we
were inspired by other studies in HCI where participants
made ratings based on images. For example, Lindgaard
et al. (2006) used images of websites that were shown
to the participants through a PC for 50–500 milliseconds.
Their aim was to study how fast people shape a judge-
ment for a website’s visual appeal. Tractinsky (1997)
used a projector and collectively showed images of differ-
ent ATM layouts to his participants, while asking them
to rate their perceived usability and beauty. Hoegg,
Alba, and Dahl (2010) used images of cookware and elec-
tric mixers that varied aesthetically to test the belief that
‘what is beautiful is good’. In a similar manner to these
previous studies, we chose as evaluation objects mobile
devices and we created a website, which on the left side
showed an image of a mobile device and on the right
side listed the questionnaire items. Using images also
allowed us to experimentally control for external par-
ameters. The included 13 mobile devices were of the
same colour, were presented without any indication of
their brand, and had their screens turned off to exclude
any effect from the operating system (Figure 5).
A total of 2239 MTurk workers participated in our
studies, used our website, and filled in the two established
cool questionnaires (Bruun et al. 2016; Sundar et al. 2014)
Figure 5. The 13 mobile devices used in the studies.
BEHAVIOUR & INFORMATION TECHNOLOGY 7

and the three established UX questionnaires (Lavie and
Tractinsky 2004; Quinn and Tran 2010; Van Schaik, Has-
senzahl, and Ling 2012). All questionnaire items were
presented randomly and were rated on a 7-point Likert
scale from ‘strongly disagree’to ‘strongly agree’.Each
participant was asked to assess only one mobile device
and participated in one EFA or CFA study. Participants
were paid an incentive ranging between 0.25$ and 0.35
$, which was in line with MTurk’s guidelines on how to
fairly pay them. From the 2239 participants, we removed
responses where participants had a considerably lower
completion time than the average, which is in line with
Kittur, Chi, and Suh (2008). We also removed all partici-
pants who reported prior experience with the mobile
device they were asked to evaluate, as prior experience
may significantly affect UX factors, and in particular per-
ceived usability (Sauro 2011). This left us with a sample of
1790 participants that was balanced in relation to gender
(892 females), had a large variety of age groups (18–72
years old, M= 33.6, SD = 10.8), and included a variety
of races (self-identified as Caucasian, African, Hispanic,
Asian, Arab, etc.). In all, 1251 of them participated in
the EFAs and 539 in the CFAs, with an average of
156.4 participants per device in the EFAs and 107.8 par-
ticipants per device in the CFAs.
6. Study 1: Convergence of coolness factors
(H1)
In this section, we present how we addressed our first
research hypothesis (H1) by examining the convergence
of factors from the two cool questionnaires.
6.1. EFA –Exploring the coolness factor model
To study H1, we asked 822 participants to rate one
mobile device each and we included three devices (274
participants per device). All participants rated the device
based on the 56 question items, that is, all items from all
questionnaires (EFA1, Table 6).
Initially we examined the reliability of each individual
factor. With the exception of pragmatic quality, all fac-
tors had exceptional Cronbach αscores, indicating
that, for the case of mobile devices, all individual factors
can be reliably measured through their respective items
(Table 7).
To define the factor model of the two combined cool-
ness questionnaires, we relied on the EFA. Combined,
the coolness questionnaires (Bruun et al. 2016; Sundar
et al. 2014) consist of 31 of the total 56 items (with the
remaining 25 representing items from the other estab-
lished UX questionnaires). We removed items by apply-
ing the cut-off criteria (low communalities (<.5) and low
factor loadings (<.65)), and in the end 18 items
remained. The final four-factor structure was identified
through Scree tests and Parallel Analysis (Monte Carlo
PCA, Watkins). This model explained 74.84% of the var-
iance, KMO was .930, and the Bartlett Test of Sphericity
was significant.
Factor A contains items that emerged from attractive-
ness (Sundar et al. 2014) and Factor B mainly from rebel-
liousness (Bruun et al. 2016). Items from originality and
subculture (Sundar et al. 2014) mostly converged on
rebelliousness, but were removed using the cut-off cri-
teria. Factor C is about usability (Bruun et al. 2016),
while Factor D deals with desirability (Bruun et al.
2016). Details are presented in Table 8.
6.2. CFA1 –Confirming the coolness factor model
In order to confirm the four-factor structure of coolness
as emerged from EFA1, we conducted a CFA study. This
included the 18 items that emerged from EFA1, and 206
participants rated one device each (Table 9).
In the first run, all indices suggested acceptable values;
that is, it was not necessary to go through modification
indices to increase model fit. Table 10 shows the respect-
ive item loadings and model-fit indices obtained in CFA1
where all loadings are significant. We also validated the
model by examining the matrix of standardised
residuals. A model with a good fit will have residuals
centred around zero and we found none larger than
Table 6. EFA1. n= number of participants, i= number of items
used as input.
Study nDevices i
EFA1 822 Samsung Galaxy S6, Blackberry Priv, and Vertu Signature
Touch
56
Table 7. Reliability analysis of factors from all questionnaires, i=
number of items.
Questionnaire Factor i
Cronbach
α
Capturing cool (Sundar, Tamul, and
Wu 2014)
Subculture 5 .913
Attractiveness 5 .911
Originality 5 .921
Cool questionnaire (Bruun et al. 2016) Desirability 6 .902
Rebelliousness 5 .886
Perceived
usability
5 .918
Aesthetics (Lavie and Tractinsky 2004) Classic aesthetics 5 .869
Expressive
aesthetics
5 .872
Attrakdiff2 (Van Schaik, Hassenzahl,
and Ling 2012)
Hedonic quality 4 .892
Pragmatic quality 4 .705
Attractiveness (Quinn and Tran 2010) Attractiveness 5 .937
8D. RAPTIS ET AL.

±2, hereby indicating a good model fit (Schreiber et al.
2006).
Based on the model-fit indices, we found that our
data support a four-factor model representing coolness.
Table 11 shows the correlation matrix between the four
factors. The diagonal elements in bold represent the
square root of average variance extracted (AVE) and
the Cronbach αvalues (in parentheses). Since the square
roots of AVEs are bigger than all factor correlations, we
can conclude that the discriminant validity is more than
adequate. The same is the case with internal consistency
(construct reliability) with high Cronbach αvalues.
6.3. Study 1 results
In this section, we present our results in relation to the
research hypothesis H1:
H1. Coolness can be measured through the four factors
of attractiveness, desirability, perceived usability, and
originality/subculture/rebelliousness.
We confirm this hypothesis. Findings from the EFA1
and CFA1 show the existence of 18 items distributed
over four factors for measuring perceived coolness:
attractiveness, perceived usability, rebelliousness, and
desirability. Attractiveness stems exclusively from Sun-
dar et al. (2014), while perceived usability and desirabil-
ity stem from Bruun et al. (2016). The rebelliousness
factor includes items from both questionnaires. Thus,
we confirm H1 for the case of assessing the perceived
coolness of mobile devices.
7. Study 2: Convergence of coolness on UX
factors
In the previous section, we confirmed the factor struc-
ture related to coolness, where the total of six factors
from Sundar et al. (2014) and Bruun et al. (2016) con-
verged on four factors. In the following, we examine
the emerging factor structure when combining the fac-
tors identified across the coolness studies as well as the
other established UX factors related to aesthetics (Lavie
and Tractinsky 2004), attractiveness(UX) (Quinn and
Table 8. Pattern Matrix with item loadings per factor in EFA1
containing only the cool questionnaires.
Factor: A B C D
Eigenvalue: 7.361 1.780 3.518 0.813
Cronbach α:.911 .918 .912 .812
Attractiveness This device is stylish
a
.919 .016 −.051 −.062
This device is hip
a
.861 −.030 −.027 .077
This device is sexy
a
.780 .011 .027 .040
This device is hot
a
.779 .023 .026 .133
This device is on the
cutting edge
a
.730 .054 .153 .026
Perceived
usability
This device is simple to
use
b
−.065 .916 −.023 .010
This device is easy to use
b
.046 .890 .004 −.023
This device is easy to learn
b
−.055 .882 .030 −.014
This device is easy to
operate
b
.001 .882 −.017 .034
This device is effortless to
use
b
.084 .759 .004 .015
Rebelliousness This device is different
b
.062 −.002 .887 −.062
This device is outside the
ordinary
a,b
.076 .022 .861 −.019
This device is
unconventional
b
−.106 −.014 .845 .022
This device is unique
a
.181 .032 .831 −.077
This device moves against
the current
b
−.093 −.019 .824 .137
Desirability This device can make me
better
b
−.059 −.005 .043 .917
This device can make me
happy
b
.099 .089 −.088 .778
This device can make me
look in control of things
b
.234 .004 .102 .618
Sum of Squared Loadings (Total variance explained):
74.84%
Note: A = Attractiveness, B = Perceived Usability, C = Rebelliousness, D =
Desirability.
a
Originates from Sundar, Tamul, and Wu (2014).
b
Originates from Bruun et al. (2016).
Table 9. CFA1. n= number of participants, i= number of items
used as input.
Study nDevices I
CFA1 206 Apple iPhone 6s Plus, and Huawei Ascend Y530 18
Table 10. Item loadings per factor and model-fit indices for the
CFA1 study.
CFA1 –Coolness Factor Model
Attractiveness This device is stylish
a
.74
This device is hip
a
.83
This device is sexy
a
.76
This device is hot
a
.75
This device is on the cutting edge
a
.72
Perceived
usability
This device is simple to use
b
.91
This device is easy to use
b
.86
This device is easy to learn
b
.87
This device is easy to operate
b
.88
This device is effortless to use
b
.72
Rebelliousness This device is different
b
.83
This device is outside the ordinary
a,b
.84
This device is unconventional
b
.63
This device is unique
a
.79
This device moves against the current
b
.63
Desirability This device can make me better
b
.78
This device can make me happy
b
.86
This device can make me look in control of
things
b
.72
Model-fit Indices
Ratio of χ
2
to df (CMIN/df, acceptance threshold ≤3) 1.5
Normed Fit Index (NFI, acceptance threshold ≥.95) .92
Incremental Fit Index (IFI, acceptance threshold ≥.95) .97
Tucker–Lewis Index (TLI, acceptance threshold ≥.95) .96
Comparative Fit Index (CFI, acceptance threshold ≥.95) .97
Goodness-of-Fit Index (GFI, acceptance threshold ≤.95) .9
Adjusted Goodness-of-Fit Index (AGFI, acceptance threshold ≤.95) .87
Root Mean Square Error of Approx. (RMSEA, accept. threshold ≤.06) .05
p of close fit (PCLOSE, acceptance threshold > .05) .43
Note: All are within acceptable thresholds, indicating good model fit.
a
Originates from Sundar, Tamul, and Wu (2014).
b
Originates from Bruun et al. (2016).
BEHAVIOUR & INFORMATION TECHNOLOGY 9

Tran 2010), and hedonic and pragmatic quality (Van
Schaik, Hassenzahl, and Ling 2012). By examining the
convergence of these factors, we seek to test the second
set of hypotheses (H2a, H2b, and H2c). In the following,
we present our findings from two EFA studies exploring
possible factor structures describing the relation between
coolness and established UX factors.
7.1. EFA –Exploring the cool-UX factor model
We started our analysis using the same dataset as before
(EFA1, Table 6), but this time we included all 56 items
(31 from the cool questionnaires and 25 from the other
established UX questionnaires). Through an EFA, we
produced a total of five models with four- and five-factor
structures. Items were removed by applying the cut-off
criteria and only if they did not belong to any factors
in any model. In all five models, the KMO was >.925, ful-
filling the criteria for sampling adequacy and the Bartlett
Test of Sphericity was significant (<.001). In the end, the
initial 56 items were reduced to 33.
Given the remaining relatively large number of items,
we chose to conduct an additional EFA study (EFA2). In
EFA2 we had 429 additional participants rate one device
each and we included a total of 5 devices (∼86 partici-
pants per device, Table 12).
The 429 participants in EFA2 rated the remaining 33
items and a 5-factor structure was identified through
Scree tests and Parallel Analysis (Monte Carlo PCA,
Watkins). By applying the cut-off criteria, the number
of items was reduced from 33 to 22 and our final
model had a KMO of .954. Cumulatively this five-factor
model explained 79.39% of the variance. In Table 13 we
present the emerged five-factor structure after EFA2,
along with the loadings of each item on factors.
Table 11. Factor correlation matrix for CFA1. Values in bold indicate the square root of AVE and Cronbach α(in parentheses).
Attractiveness Per. Usability Rebelliousness Desirability
Attractiveness .761 (.874)
Perceived usability .35 .850 (.923)
Rebelliousness .55 .2 .750 (.861)
Desirability .74 .42 .41 .789 (.829)
Table 12. EFA2. n= number of participants, i= number of items
used as input.
Study nDevices i
EFA2 429 HTC One M8, OnePlus One, Tag Heuer Meridiist, Nokia
222, Philips Fluid
33
Table 13. Pattern matrix with item loadings per factor in EFA2.
Factor: A B C D E
Eigenvalue: 7.862 5.184 2.284 1.346 0.790
Cronbach α:.946 .933 .921 .790 .905
Hedonic I find this device: plain/eye catching
a
.909 −.003 .127 −.005 −.079
I judge this device to be: cheap/premium
b
.879 −.082 −.092 −.073 −.006
I judge this device to be: dull/captivating
b
.865 .010 −.009 .026 .123
I find this device: boring/interesting
a
.851 .015 .035 −.045 .092
I judge this device to be: unimaginative/creative
b
.767 .037 .236 −.009 .006
Perceived Usability This device is simple to use
c
−.004 .924 −.003 .002 −.033
This device is easy to use
c
−.005 .924 −.006 .010 −.018
This device is easy to operate
c
.058 .919 −.057 −.035 −.039
This device is easy to learn
c
−.106 .842 .084 −.103 .005
This device is effortless to use
c
.024 .799 −.025 .034 .118
Rebelliousness This device moves against the current
c
−.039 .106 .910 .059 −.097
This device is outside the ordinary
c,d
.057 −.009 .866 −.013 .051
This product stands apart from similar products
d
.015 −.033 .833 −.068 .116
This device is different
c
.093 −.076 .815 −.013 .046
This device is unconventional
c
.071 −.069 .802 .015 −.014
Classic Aesthetics This device has clear design
e
−.065 .043 .055 −.922 .001
This device has clean design
e
.116 .032 −.090 −.840 .016
Desirability This device can make me better
c
−.101 .035 .015 .091 .938
This device can make me look in control of things
c
−.023 −.047 .120 −.112 .810
This device can make me look good
c
.067 −.111 .086 −.127 .779
This device can make me happy
c
.138 .132 −.110 −.005 .755
This device totally connects with me
c
.289 .115 −.114 .003 .678
Sum of Squared Loadings (Total variance explained): 79.39%
A = Attractiveness, B = Perceived Usability, C = Rebelliousness, D = Classic Aesthetics and E = Desirability.
a
Originates from Quinn and Tran (2010).
b
Originates from Van Schaik, Hassenzahl, and Ling (2012).
c
Originates from Bruun et al. (2016).
d
Originates from Sundar, Tamul, and Wu (2014).
e
Originates from Lavie and Tractinsky (2004).
10 D. RAPTIS ET AL.

Throughout the EFA studies, we observed a trend on
how items converged on the five factors. Items from the
two attractiveness factors (Quinn and Tran 2010; Sundar
et al. 2014), expressive aesthetics (Lavie and Tractinsky
2004), and hedonic quality (Van Schaik, Hassenzahl,
and Ling 2012) converged around Factor A (hedonic).
Furthermore, items from pragmatic quality (Van Schaik,
Hassenzahl, and Ling 2012) converged on Factor B (per-
ceived usability) with low factor loadings, which led to
their removal. Thus, Factor B consists of items from
the perceived usability factor identified in Bruun et al.
(2016). Items from subculture and originality (Sundar
et al. 2014) and rebelliousness (Bruun et al. 2016) con-
verged on Factor C (rebelliousness). Items from the clas-
sic aesthetics factor (Lavie and Tractinsky 2004) solely
define Factor D (classic aesthetics), with no convergence
on other factors. Finally, a few specific questions from
expressive aesthetics, classic aesthetics, and attractive-
ness converged on Factor E (desirability) with low factor
loadings. Thus, Factor E is defined by items from desir-
ability (Bruun et al. 2016).
7.2. CFA –Confirming the cool-UX factor model
To confirm the proposed five-factor model from EFA2,
we had 333 participants rate one mobile device, and we
included 3 different devices in total (CFA2, Table 14),
that is, each device was assessed by 111 participants on
average. None of the participants had taken part in the
previous studies.
To obtain an acceptable model fit, we went through
four iterations where we removed one item at a time
based on the largest modification indices. After that
point all indices suggested a good model fit; that is, it
was not necessary to go through further iterations.
Through these iterations, we removed four items from
the 22 identified in EFA2. Thus, the final CFA model
consists of 18 items. Table 15 shows the respective
item loadings and model-fit indices obtained in CFA2
where all loadings are significant. We also validated the
model by examining the matrix of standardised
residuals. A model with a good fit will have residuals
centred around zero and we found none larger than
±2, hereby indicating a good model fit (Schreiber et al.
2006).
Table 16 presents the correlation matrix between the
five factors, which shows that none of the factors have
a1–1 correlation. The diagonal elements in bold
represent the square root of AVE as well as the Cronbach
αvalues (in parentheses). Since the elements exceed all
factor correlations except one, discriminant validity is
adequate. That said, the hedonic and desirability factors
do have a correlation of .79, which indicates that these
factors are closely related and one can be used to predict
the other. Nevertheless, all factors are consistently separ-
ated throughout our EFA and CFA studies, that is, they
are measuring different UX factors. Furthermore, in
relation to internal consistency (construct reliability),
Cronbach αvalues are high, which shows that the
items can reliably measure the five factors.
7.3. Study 2 results
Our second hypothesis related coolness to other estab-
lished UX factors and is divided into three parts (H2a,
H2b, and H2c). We will address each hypothesis
individually.
H2a. The coolness factor of attractiveness(cool) con-
verges on established UX factors of classic/expressive
aesthetics/hedonic quality/attractiveness(UX).
Table 14. CFA2. n= number of participants, i= number of
question items used as input.
Study nDevices i
CFA2 333 Nexus 6P, North Face M8, and Blackberry Classic 22
Table 15. Item loadings per factor and model-fit indices for the
CFA2 study.
CFA2 –Cool-UX Factor Model
Hedonic I find this device: plain/eye catching
a
.89
I judge this device to be: dull/captivating
b
.88
I find this device: boring/interesting
a
.90
I judge this device to be: unimaginative/
creative
b
.84
Perceived
Usability
This device is simple to use
c
.87
This device is easy to use
c
.88
This device is easy to operate
c
.91
This device is easy to learn
c
.83
Rebelliousness This device moves against the current
c
.7
This device is outside the ordinary
c,d
.86
This product stands apart from similar products
d
.79
This device is different
c
.87
Classic aesthetics This device has clear design
e
.73
This device has clean design
e
.76
Desirability This device can make me better
c
.78
This device can make me look in control of
things
c
.74
This device can make me look good
c
.86
This device can make me happy
c
.78
Model-fit Indices
Ratio of χ
2
to df (CMIN/df, acceptance threshold ≤3) 1.9
Normed Fit Index (NFI, acceptance threshold ≥.95) .95
Incremental Fit Index (IFI, acceptance threshold ≥.95) .98
Tucker–Lewis Index (TLI, acceptance threshold ≥.95) .97
Comparative Fit Index (CFI, acceptance threshold ≥.95) .97
Goodness-of-Fit Index (GFI, acceptance threshold ≤.95) .93
Adjusted Goodness-of-Fit Index (AGFI, acceptance threshold ≤.95) .9
Root Mean Square Error of Approx. (RMSEA, accept. threshold ≤.06) .05
pof close fit (PCLOSE, acceptance threshold > .05) .42
Note: All are within acceptable thresholds, indicating good model fit.
a
Originates from Quinn and Tran (2010).
b
Originates from Van Schaik, Hassenzahl, and Ling (2012).
c
Originates from Bruun et al. (2016).
d
Originates from Sundar, Tamul, and Wu (2014).
e
Originates from Lavie and Tractinsky (2004).
BEHAVIOUR & INFORMATION TECHNOLOGY 11

This hypothesis is not supported. As in previous
studies (Diefenbach, Kolb, and Hassenzahl 2014; Has-
senzahl and Monk 2010), our results showed that expres-
sive aesthetics and hedonic quality converge. We also
identified that these also converged with the two attrac-
tiveness factors (Quinn and Tran 2010; Sundar et al.
2014). At the same time, classic aesthetics (Lavie and
Tractinski 2004) formed an independent factor. Thus,
even though we expected all five factors to be merged
into one, they merged into two: A) a hedonic factor
which consisted of two question items from Quinn and
Tran’s(2010) attractiveness factor, and two items from
Van Schaik, Hassenzahl, and Ling’s(2012) hedonic qual-
ity factor, and B) a classic aesthetics factor that emerged
solely from Lavie and Tractinsky (2004). For these
reasons, we falsify hypothesis H2a.
H2a. The coolness factor of perceived usability con-
verges on the established UX factor of pragmatic quality.
Table 15 shows that the emerged perceived usability
factor consists exclusively of question items stemming
from the perceived usability factor as suggested by
Bruun et al. (2016). This happened because even though
all items from Van Schaik, Hassenzahl, and Ling’s(2012)
pragmatic quality factor consistently followed the per-
ceived usability items, they all had loadings below the
cut-off level. Additionally, since we did not observe any
of the pragmatic items converging on any other factor,
we verify hypothesis H2b.
H2a. The coolness factors of originality/subculture/
rebelliousness and desirability do not converge on any
of the established UX factors.
Finally, in relation to H2c, both factors of rebellious-
ness and desirability that emerged during the EFA
studies were retained in the five-factor model from
CFA2. Items from originality and subculture did con-
verge on the rebelliousness factor, but with lower load-
ings. Thus, hypothesis H2c is also verified.
8. Discussion
Our purpose with this paper was twofold. Firstly, we
wanted to make a systematic comparison of the under-
lying factors between the recently proposed question-
naires for measuring perceived coolness. Thus, in our
first study, we examined the extent of convergence
between the suggested coolness factors and we demon-
strated that there are differences as well as overlaps. Sec-
ondly, we also wanted to position the coolness factors in
relation to established UX ones and thus contribute to
the dimensionality explosion challenge (Bargas-Avila
and Hornbæk 2011). Through our research effort in
our second study, we managed to combine all factors
into a single model, thus we strengthened the position
of coolness within UX research. In the following subsec-
tions, we discuss the implications for research and
practice.
8.1. Implications for UX research
When studying the two cool questionnaires (Bruun et al.
2016; Sundar et al. 2014), we produced a model that nar-
rowed down the initial six factors into four (Table 8:
attractiveness, perceived usability, rebelliousness, and
desirability). When we followed the same process by
including established UX questionnaires, a new model
emerged that contained not 11, but 5 factors (Table 15:
hedonic quality, classic aesthetics, desirability, perceived
usability, and rebelliousness). What we believe is inter-
esting for our research community is to understand the
relation among these factors.
In this part of the discussion, we contribute to under-
standing the relation among these factors by linking back
to theory. According to the literature, when people
observe a product (or a person), they immediately
make a judgement on its overall coolness (Pountain
and Robbins 2000). When there is no actual usage with
a product, as was the case in our study, this judgement
is initially based on the externally observable aesthetic
attributes (outer coolness), which people use to infer a
judgement of personality characteristics (inner coolness).
Then, both inner coolness and outer coolness shape the
overall judgement of coolness (Bruun et al. 2016; Poun-
tain and Robbins 2000; Raptis, Kjeldskov, and Skov
2013). Figure 6 shows the theoretical relation among per-
ceived inner coolness, perceived outer coolness, and
overall coolness judgement.
We took this theoretical model one step further by
including the emerged five factors of hedonic quality,
classic aesthetics, desirability, usability, and
Table 16. Factor correlation matrix for CFA2. Diagonal values in bold indicate: the square root of AVE and Cronbach α(in parentheses).
Hedonic Per.Usability Rebelliousness Classic Aesthetics Desirability
Hedonic .878 (.929)
Per. Usability .29 .873 (.927)
Rebelliousness .66 .14 .806 (.878)
Classic Aesthetics .47 .66 .14 .745 (.712)
Desirability .79 .41 .48 .55 .791 (0.868)
12 D. RAPTIS ET AL.

rebelliousness. We hypothesise a UX inference model in
a similar manner as Hassenzahl and Monk (2010) and
Van Schaik et al. (2012), who showed what we perceive
as beautiful is also perceived as good, which in turn is
perceived as usable, that is, ‘What is beautiful is good
and what is good is usable’. In order to validate our pro-
posed inference model, we used inferential statistics
(Partial Least Squares, Vinzi et al. 2010). Based on our
analysis, we argue that hedonic quality and classic aes-
thetics contribute to outer coolness, since they both
relate to the aesthetic attributes of a product, while per-
ceived usability, rebelliousness, and desirability contrib-
ute to inner cool. In the Appendix, the final validated
inference model is presented along with standardised
regression coefficients, T-statistics, the percentage of
explained variance, as well as details on the process fol-
lowed. In the following figure, we present a simplified
version of this model that depicts how the emerged
five factors are clustered around inner cool and outer
cool, and how they both contribute to the overall cool-
ness judgement.
The proposed inference model of users’experiences
with mobile devices through outer, inner, and overall
cool (Figure 7) is a valid and useful tool for researchers.
Firstly, it demonstrates the existence of an inference rule:
‘The perception of product aesthetics influences per-
ceived product personality characteristics, and both
shape the overall coolness judgement’. Secondly, it
shows that people do infer inner cool from outer cool
when they believe that it is a relevant rule for the situ-
ation (e.g. during first impression with mobile devices,
i.e. without actual usage). Thirdly, it demonstrates that
overall coolness cannot be inferred only by factors
related to externally observable attributes. Inner coolness
is also needed. Fourthly, our inference model strongly
positions coolness within UX research, and it can be
used to explain and/or predict the relationship among
the five emerged factors.
Another important implication of our research work is
that we showed that the established UX factors we use in
our research community converge. This leads to some fac-
tors being more prevalent than others, while others con-
verge, for example, pragmatic quality and perceived
usability. This reduces the number of factors that are rel-
evant to consider. This finding taps directly into the UX
community discussion of the dimensionality explosion.
Bargas-Avila and Hornbæk (2011), for instance, mention:
‘the main problem of this dimensionality explosion is that
the relation to established constructs is rarely made clear’.
Our study provides a way of dealing with this through the
proposed measurement model for the case of assessing
perceived UX of mobile devices.
Of course, further research is needed for other pro-
ducts than mobile devices, for different contexts of use
(e.g. after long-term usage), and for different cultural
groups, in order to test the applicability of inferring
inner-cool from outer-cool rule and its performance
(e.g. is the direction of the effect always the same?). Fur-
thermore, we find it of particular interest to compare our
inference model with existing ones, (e.g. Van Schaik,
Hassenzahl, and Ling 2012), as such comparisons may
provide answers to important research questions such
as: Are these models applicable in all contexts? For all
products? Furthermore, an important research activity
that we believe our community needs to pursue is to con-
tinue the study of convergence of the rest of the UX fac-
tors. Do other established UX factors that we did not
include in our studies (such as pleasure) converge to
the five we managed to identify in this paper? Are
there other influential UX factors which are currently
unknown?
Finally, we believe there is a need for more research on
the relation between hedonic quality and classic aes-
thetics. Since both of them emerged as unique and
Figure 6. Theoretical relationships among perceived inner cool,
perceived outer cool, and overall coolness judgement.
Figure 7. Simplified inference model showing that hedonic qual-
ity and classic aesthetics cluster around outer cool, while desir-
ability, rebelliousness, and perceived usability cluster around
inner cool. Both perceived outer and inner cool shape the overall
coolness judgement. All paths are significant (details in the
Appendix).
BEHAVIOUR & INFORMATION TECHNOLOGY 13

distinct factors and both measure aesthetic attributes, it
is crucial to understand what they actually measure.
We consider this challenge as important since it was
also identified in previous research work (Bruun et al.
2016). Based on our findings, we propose that the two
items of classic aesthetics that remained in our study
(clean/clear design) are related to the cognitive process
of recognition, while the four hedonic items relate to
an intentional (or even unintentional) evaluation process
of a product’s aesthetic appeal that occurs afterwards. As
an example, we believe our participants recognised the
product they experienced in our study as a mobile device,
and then they evaluated its appeal. More studies with this
emphasis are needed in order to verify our assumption
and we consider them as important as they may shed
light on the cognitive mechanisms people use to aesthe-
tically evaluate our produced designs.
8.2. Implications for UX practice
Our findings also have implications for practice. Firstly,
we reconfirmed that all five established questionnaires
we included in our studies can reliably measure their per-
tinent factors for the case of perceived UX of mobile
devices (Table 7). The interesting issue though for prac-
titioners is how to use these questionnaires in practice.
If practitioners want to measure a specific perceived
UX factor, that is, the perceived usability of a produced
design, then any of these questionnaires that measures
perceived usability can be used, as it will provide reliable
results. A challenge though exists if the purpose is to
have a holistic evaluation of a product’s perceived attri-
butes. In such cases, practitioners should administer
combinations of all these questionnaires in efforts to
assess various aspects of UX. However, measuring all
UX factors would mean that participants will have to
answer a relatively large set of questions (56 in case of
the questionnaires included in this study alone). Our
findings help practitioners deal with dimensionality
explosion and the large number of question items
through the proposed five-factor model (Figure 7) and
its items (Table 15). We demonstrated that the two
established UX factors that converge on outer coolness
(hedonic quality and classic aesthetics) can be measured
using six items. Inner coolness can be measured through
11 items from the Cool Questionnaire (Bruun et al. 2016)
and 1 item from the Capturing Cool questionnaire (Sun-
dar et al. 2014). Finally, the overall coolness judgement
can be measured by three items (details on how they
emerged can be found in the Appendix). Thus, instead
of answering 56 items which are related to several per-
ceived UX factors that practitioners do know that they
overlap, participants can answer the more manageable
21 items, which belong to factors that are independent.
The final questionnaire that measures both perceived
inner and outer coolness and the overall coolness judge-
ment as well as their resulting items is shown in Figure 8.
Furthermore, we argue that our questionnaire/model
will be more useful to practitioners if combined with
qualitative methods. For example, if a product scores
low on rebelliousness and this has a negative impact
on its overall coolness, then practitioners can through,
for example, in-depth interviews, identify specific design
elements in relation to rebelliousness that need to be
changed, produce redesigns, and then re-evaluate them.
Thus, coolness becomes an essential design criterion,
which not only can be measured, but also understood
in relation to other established UX factors.
Finally, we define three challenges for our combined
questionnaire/model that practitioners need to be
aware of, which should be researched more in order to
increase its applicability. The first is related to its per-
formance. At this moment, we do not know what it
means for a product to score, for example, 5 on usability
or 3.5 on rebelliousness; that is, we do not know if such
scores are good or bad for a particular product. In order
to understand the behaviour of the model, research
approaches that were used for other questionnaires in
the past should be applied. For example, for the System
Usability Scale (SUS) scale, Bangor, Kortum, and Miller
(2008) conducted a meta-analysis of previous studies,
and concluded on the meaning of an SUS score in
relation to a product’s usability. Secondly, since coolness
is deeply rooted in the cultural communities people
belong to (O’Donnell and Wardlow 2000), we need to
test the model’s behaviour in different communities.
For example, we may have different results not only in
Asian, or European, cultures, but also within different
subcultures. Finally, we need to study the model’s behav-
iour in relation to time. For example, it is known that
users who interact with products for long periods of
time change their perception of usability (Sonderegger
et al. 2012). To what extent do the rest of the factors
have similar behaviour? Such knowledge can be extre-
mely useful for practitioners as it will allow them to
understand how users’experiences develop over time,
hereby leading to decisions, for example, on when to
redesign products.
9. Conclusions
In this paper, we explored how a large set of cool and UX
factors converge. Our paper contributes to the dimen-
sionality explosion discussion (Bargas-Avila and Horn-
bæk 2011) in two ways. Firstly, we showed that the
two existing questionnaires for measuring perceived
14 D. RAPTIS ET AL.

coolness converge on four factors (Table 8). Secondly, we
established coolness within UX research by comparing it
against established UX factors. Our resulting model
shows that eleven distinct cool and UX factors (Table
7) converge on five (Figure 7 and Table 15).
Our research identified a number of implications for
researchers and practitioners. In relation to UX research,
first, we positioned coolness within UX by demonstrat-
ing how it relates to established UX factors. Additionally,
we moved a step further by proposing an inference
model, which is based on the emerged five factors and
has a strong theoretical foundation that distinguishes
between outer cool (the perceived aesthetic character-
istics of a product) and inner cool (the perceived person-
ality characteristics of a product). The model can be used
to explain and/or predict people’s judgemental mechan-
isms in relation to perceived coolness of mobile devices.
In relation to practice, firstly, we demonstrated that
each of the deployed questionnaires in our two studies
is reliable. Secondly, we compared the two cool question-
naires and produced a valid tool for measuring perceived
inner coolness (Table 10). Finally, we produced a ques-
tionnaire (Figure 8) that measures both perceived inner
and perceived outer coolness through 5 distinct factors
with 18 items, and 3 evaluative items for the overall
cool judgement. Both the model and the final question-
naire can be used to holistically assess users’experiences
with mobile devices.
One of the limitations of our study is that participants
were asked to evaluate images and not the actual mobile
Figure 8. Final perceived inner cool and outer cool questionnaire. Measured inner cool and outer cool factors and their items as well as
the three evaluative items for the overall coolness judgement. The word ‘device’can be replaced with a suitable product.
BEHAVIOUR & INFORMATION TECHNOLOGY 15

phones. As others have also used this approach, it would
be interesting to investigate if the use of images as test
objects leads to differences in results when compared
to the use of physical artefacts. Furthermore, we believe
that both our model and its resulting questionnaire are
applicable to other types of digital artefacts, as it was
the case with most established UX questionnaires (such
as Attrakdiff2), by replacing the word ‘device’with the
product under evaluation. Nevertheless, further research
is needed to test them in different conditions and for
different products, to verify their generalisability and
applicability.
Note
1. Tools for deploying the COOL questionnaire as well as
for analysing collected data can be found in: http://
thecoolquestionnaire.weebly.com/
Acknowledgements
We would like to thank Dr Veronica Hinkle for her invaluable
help as well as all the people who participated in our studies.
References
Bangor, A., P. Kortum, and J. Miller. 2008.“An Empirical
Evaluation of the System Usability Scale.”International
Journal of Human-Computer Interaction 24 (6): 574–594.
Bargas-Avila, J. A., and K. Hornbæk. 2011.“Old Wine in New
Bottles or Novel Challenges? A Critical Analysis of
Empirical Studies of User Experience.”In Proceeding CHI
2011, 2689–2698. New York: ACM.
Boujarwah, F., G. Abowd, and R. Arriaga. 2012.“Socially
Computed Scripts to Support Social Problem Solving
Skills.”In Proceeding. CHI 2012, 1987–1996. New York:
ACM.
Bruun, A., D. Raptis, J. Kjeldskov, and M. B. Skov. 2016.
“Measuring the Coolness of Interactive Products: The
COOL Questionnaire.”Behaviour & Information
Technology 35 (3): 233–249.
Bulmer, M. G. 1979.Principles of Statistics. New York: Courier
Dover.
Chin, W. W. 1998.“The Partial Least Squares Approach to
Structural Equation Modelling.”In Modern Methods for
Business Research, edited by G. A. Marcoulides, 295–336.
Mahwah, NJ: Lawrence Erlbaum.
Chin, W. W., B. L. Marcolin, and P. N. Newsted. 2003.“A
Partial Least Squares Latent Variable Modeling Approach
for Measuring Interaction Effects: Results from a Monte
Carlo Simulation Study and an Electronic-mail Emotion/
Adoption Study.”Information Systems Research 14 (2):
189–217.
Conan, N. 2008.Behind the Viral Video: What’s Fake, What’s
Real. National Public Radio.
Culén, A. L., and A. A. Gasparini. 2012.“Situated Techno-
Cools: Factors that Contribute to Making Technology Cool
in a Given Context of Use.”PsychNology 10 (2): 117–139.
Dar-Nimrod, I., I. G. Hansen, T. Proulx, D. R. Lehman, B. P.
Chapman, and P. R. Duberstein. 2012.“Coolness: An
Empirical Investigation.”Journal of Individual Differences
33 (3): 175–185.
Diefenbach, S., N. Kolb, and M. Hassenzahl. 2014.“The
‘Hedonic’in Human-Computer Interaction –History,
Contributions, and Future Research Directions.”In
Proceeding. DIS 2014, 305–314. New York: ACM Press.
Falk, R. F., and N. B. Miller. 1992.A Primer for Soft Modelling.
Akron, OH: The University of Akron Press.
Field, A. 2009.Discovering Statistics Using SPSS. London: SAGE.
Gioia, T. 2009.The Birth and Death of the Cool. Golden, CO:
Speck Press.
Hassenzahl, M., M. Burmester, and F. Koller. 2003.“Attrakdiff:
Ein Frageboten zur Messung wahrgenommener hedo-
nischer und pragmatischer Qualität.”In Mensch &
Computer 2003. Interaction in Bewegung, 187–196. Bonn:
Gesellshaft für Informatik.
Hassenzahl, M., and A. Monk. 2010.“The Inference of
Perceived Usability from Beauty.”Human-Computer
Interaction 25 (3): 235–260.
Heer, J., and M. Bostock. 2010.“Crowdsourcing Graphical
Perception: Using Mechanical Turk to Assess
Visualization Design.”In Proceeding. CHI 2010, 203–212.
New York: ACM.
Heimerl, K., B. Gawalt, K. Chen, T. Parikh, and B. Hartmann.
2012.“Community Sourcing: Engaging Local Crowds to
Perform Expert Work Via Physical Kiosks.”In Proceeding,
CHI 2012, 1539–1548. New York: ACM.
Henseler, J., C. Ringle, and R. Rayner. 2009.“The Use of Partial
Least Squares Modelling in International Marketing. In New
Challenges in International Marketing, edited by T.
Cavusgil, R. Sinkovicks, and P. Ghauri, 277–319. Bingley:
Emerald.
Hoegg, J. A., J. W. Alba, and D. W. Dahl. 2010.“The Good, the
Bad, and the Ugly: Influence of Aesthetics on Product
Feature Judgments.”Journal of Consumer Psychology 20
(4): 419–430.
Holtzblatt, K. 2011.“What makes things cool?.”Interactions
18 (6): 40–47.
Horton, M., J. C. Read, D. Fitton, N. Toth, and L. Little. 2012.
“Too Cool at School-understanding Cool Teenagers.”
Psychnology 10 (2): 73–91.
Huff, C., and D. Tingley. 2015.“Locke and the Nature of
Political Authority.”The Review of Politics 2(3):1–22.
Janlert, L. E., and E. Stolterman. 1997.“The Character of
Things.”Design Studies 18 (3): 297–314.
Jordan, P. W. 1997.“Products as Personalities.”In
Contemporary Ergonomics, edited by M. A. Hanson, 449–
453. Abingdon: Taylor & Francis.
Jöreskog, K. G., and H. Wold. 1982.“The ML and PLS
Techniques for Modelling with Latent Variables:
Historical and Comparative Aspects.”In Systems Under
Indirect Observation: Causality, Structure, Prediction. Vol.
1, edited by H. Wold and K. G. Jöreskog, 263–270.
Amsterdam: North-Holland.
Kerner, N., G. Pressman, and A. Essex. 2007.Chasing Cool:
Standing out in Today’s Cluttered Marketplace. New York:
Atria.
Kittur, A., E. H. Chi, and B. Suh. 2008.“Crowdsourcing User
Studies with Mechanical Turk.”In Proceeding. CHI 2008,
453–456. New York: ACM Press.
16 D. RAPTIS ET AL.

Kutner, M. H., C. J. Nachtsheim, and J. Neter. 2004.Applied
Linear Regression Models. 4th ed. Irwin: McGraw-Hill.
Lavie, T., and N. Tractinsky. 2004.“Assessing Dimensions of
Perceived Visual Aesthetics of Web Sites.”International
Journal of Human-Computer Studies 60: 269–298.
Law, E. L.-C., V. Roto, M. Hassenzahl, A. Vermeeren, and J.
Kort. 2009.“Understanding, Scoping and Defining User
Experience: A Survey Approach.”In Proceeding. CHI,
719–728. Boston: ACM Press.
Levy, S. 2006.The Perfect Thing: How the iPod Shuffles
Commerce, Culture, and Coolness. New York: Simon &
Schuster.
Lindgaard, G., G. Fernandes, C. Dudek, and J. Brown. 2006.
“Attention Web Designers: You Have 50 Milliseconds to
Make a Good First Impression!.”Behaviour &
Information Technology 25 (2): 115–126.
Lowry, P. B., and J. Gaskin. 2014.“Partial Least Squares (PLS)
Structural Equation Modeling (SEM) for Building and
Testing Behavioral Causal Theory: When to Choose it and
How to Use It.”IEEE Transactions on Professional
Communication 57 (2): 123–146.
MacAdams, L. 2001.Birth of the Cool. Beat, Bebop, and the
American Avant-Garde. New York: The Free Press.
McCrickard, D. S., J. Barksdale, and F. Doswell. 2012.
“Understanding Cool: An Analytic Exploration of
Contributing Factors for Teens.”PsychNology 10 (2): 93–102.
Nancarrow, C., P. Nancarrow, and J. Page. 2002.“An Analysis
of the Concept of Cool and Its Marketing Implications.”
Journal of Consumer Behaviour 1 (4): 311–322.
O’Donnell, K. A., and D. L. Wardlow. 2000.“A Theory on the
Origins of Coolness.”Advances in Consumer Research 27:
13–18.
OECD. 2016.Population with Tertiary Education. doi:10.
1787/0b8f90e9-en
Pountain, D., and D. Robbins. 2000.Cool Rules. Anatomy of an
Attitude. London: Reaction Books.
Quinn, J. M., and T. Q. Tran. 2010.“Attractive Phones Don’t
Have to Work Better: Independent Effects of Attractiveness,
Effectiveness, and Efficiency on Perceived Usability.”In
Proceeding. CHI 2010, 353–362. New York: ACM Press.
Raptis, D., J. Kjeldskov, and M. B. Skov. 2013.“Understanding
‘Cool’in Human-Computer Interaction Research and
Design.”In Proceeding. OzCHI 2013, 53–62. New York:
ACM Press.
Read, C. R., M. Horton, and D. Fitton. 2012.“Being Cool-get-
ting Persona-l.”CHI Workshop on Cool aX Continents,
Cultures and Communities 2012.
Reinecke, K., and A. Bernstein. 2011.“Improving
Performance, Perceived Usability, and Aesthetics with
Culturally Adaptive User Interfaces.”ACM Transactions
on Computer-Human Interaction 18 (2): 1–29. Article
No. 8.
Ringle, C. M., S. Wende, and A. Will. 2006.SmartPLS 2.0.
Hamburg: University of Hamburg.
Ross, J., L. Irani, M. S. Silberman, A. Zaldivar, and B.
Tomlinson. 2010.“Who Are the Crowdworkers?: Shifting
Demographics in Mechanical Turk.”In Proceeding. CHI
EA 2010, 2863–2872. New York: ACM Press.
Sauro, J. 2011. Does Prior Experience Affect Perceptions of
Usability? http://www.measuringu.com/blog/prior-
exposure.php.
Schreiber, J. B., A. Nora, F. K. Stage, E. A. Barlow, and J. King.
2006.“Reporting Structural Equation Modeling and
Confirmatory Factor Analysis Results: A Review.”The
Journal of Educational Research 99 (6): 323–338.
Sobel, M. E. 1982.“Asymptotic Confidence Intervals for
Indirect Effects in Structural Equation Models.”
Sociological Methodology 13: 290–312.
Sonderegger, A., G. Zbinden, A. Uebelbacher, and J. Sauer.
2012.“The Influence of Product Aesthetics and Usability
Over the Course of Time: A Longitudinal Field
Experiment.”Ergonomics 55 (7): 713–730.
Sundar, S. S., D. J. Tamul, and M. Wu. 2014.“Capturing ‘Cool’:
Measures for Assessing Coolness of Technological
Products.”International Journal of Human-Computer
Studies 72: 169–180.
Tabachnick, B. G., and L. S. Fidell. 2013.Using Multivariate
Statistics. 6th ed. New Jersey: Pearson Education.
Tenenhaus, M., E. V. Vinzi, Y. M. Chatelin, and C. Lauro.
2005.“PLS Path Modeling.”Computational Statistics &
Data Analysis 48 (1): 159–205.
Tractinsky, N. 1997.“Aesthetics and Apparent Usability:
Empirically Assessing Cultural and Methodological
Issues.”In Proceeding. CHI 1997, 115–122. New York:
ACM Press.
Van Schaik, P., M. Hassenzahl, and J. Ling. 2012.“User-
Experience from an Inference Perspective.”ACM
Transactions on Computer-Human Interaction 19 (2): 1–
25. Article No. 11.
Vermeeren, A. P. O. S., E. L. C. Law, V. Roto, M. Obrist, J.
Hoonhout, and K. Väänänen-Vainio-Mattila. 2010.“User
Experience Evaluation Methods: Current State and
Development Needs.”In Proceeding. NordiCHI 2010,
521–530. New York: ACM Press.
Vinzi, V. E., W. W. Chin, J. Herseler, and H. Wang. 2010.
Handbook of Partial Least Squares: Concepts, Methods and
Applications in Marketingand Related Fields. Berlin: Springer.
Watkins, M. W. Monte Carlo PCA for Parallel Analysis.
Accessed http://edpsychassociates.com/Watkins3.html
Wilson, B. 2010.“Using PLS to Investigate Interaction Effects
Between Higher Order Brand Constructs.”In Handbook of
Partial Least Squares: Concepts, Methods and Applications
in Marketing and Related Fields, edited by V. Esposito
Vinzi, et al. Berlin: Springer.
BEHAVIOUR & INFORMATION TECHNOLOGY 17

Appendix: Developing the PLS inner cool–outer cool inference model
In order to test the inferential relationship between outer cool, inner cool, and the overall coolness judgement, we applied a Partial
Least Squares (PLS) analysis (Vinzi et al. 2010). This is an ideal technique for models with high complexity and when a theoretical
understanding of a domain needs to be tested (Jöreskog and Wold 1982; Falk and Miller 1992).
In order to have enough data to conduct PLS, we merged the data from EFA1, EFA2, and CFA2 (1584 participants) and we
focused only on final 18 items that emerged from the CFA2 (Table 15). These had to be compared with an overall coolness judge-
ment. Bruun et al. (2016) measured overall coolness judgement through the item ‘This device is cool’, while Sundar, Tamul, and
Wu (2014) did the same using the same item and eight additional ones. All nine of them were measured throughout all EFAs and
CFAs. In order to narrow down their number, we conducted a reliability analysis on the overall coolness judgement, which resulted
in three items: ‘When I think of cool things, devices like this come to mind’,‘This device is cool’, and ‘If I made a list of cool things,
this device would be on it’(Cronbach α= .904).
The dataset was analysed using the SmartPLS v2.0 software (Ringle, Wende, and Will 2006). Since we had two second-level
formative factors (perceived inner and outer cool), we used the hierarchical components approach, which is the most popular
when estimating higher order factors with PLS (Chin, Marcolin, and Newsted 2003; Tenenhaus et al. 2005; Wilson 2010). PLS
produces the standardised regression coefficients (path estimates) between factors and we assessed the significance of path coeffi-
cients through bootstrap analysis (with N= 5000, as proposed by Henseler, Ringle, and Rayner 2009).
In the first step, we applied the standardised regression coefficients between desirability, perceived usability, and rebelliousness
to reflect the inner cool factor, followed by the coefficients between hedonic and classic aesthetics to reflect the outer cool factor.
The second step in the process was to analyse coefficients between the inner cool and outer cool factors on the overall coolness
judgement for a mobile device. The final inference model can be found in the following figure.
Two parameters are usually applied for assessing the goodness of such models: the strength of each path coefficient and the
percentage of explained variance (R
2
). All path coefficients in our model were statistically significant (p< .001) and we had
one substantial and one moderate R
2
value (Chin 1998). As a last step in the process, we tested the significance of the mediation
effect of inner cool, that is, whether inner cool could be excluded from the model, through a Sobel test (Sobel 1982) as rec-
ommended by Lowry and Gaskin (2014). The Sobel test value (6.206) was statistically significant (p< .001), which means that
inner cool partially mediates outer cool in determining the overall coolness judgement of a product, and thus it cannot be ignored.
Figure A1. (A) PLS measurement and structural model for the first-order formative constructs of outer cool and inner cool with factor
loadings per item and (B) PLS structural model for the second-order formative constructs of inner and outer cool and the overall judge-
ment about the coolness of a mobile device. Values in parentheses indicate effects without inner cool. βstands for standardised
regression coefficients, tfor T-statistic, and R
2
for percentage of explained variance. ***p< .001.
18 D. RAPTIS ET AL.