Content uploaded by Tobias Kowatsch
Author content
All content in this area was uploaded by Tobias Kowatsch
Content may be subject to copyright.
TOWARDS EMPIRICALLY VALIDATED UBIQUITOUS
INFORMATION SYSTEMS: RESULTS FROM A PRETEST
AND THREE EMPIRICAL STUDIES
Kowatsch, Tobias, Institute of Technology Management (ITEM-HSG), University of
St.Gallen, Dufourstrasse 40a, CH-9000 St.Gallen, Switzerland, tobias.kowatsch@unisg.ch
Maass, Wolfgang, Chair in Business Administration esp. Information and Service Systems,
Saarland University, 66123 Saarbrücken, Germany, wolfgang.maass@iss.uni-saarland.de
Abstract
Research on ubiquitous information systems (UIS) has recently gained attention in the IS community.
But dedicated empirical instruments that are robust and capture individual characteristics of UIS are
still missing. A rather new empirical construct derived from Task-Technology Fit theory was proposed
and denoted as Situation-Service Fit (SSF) (Maass et al., 2012). This paper shows how SSF can be
used within the Situational Design Method for IS, a design method tailored to UIS. For this purpose,
the SSF instrument is first tested for face validity as well as discriminant and convergent validity.
Then, SSF is evaluated as a predictor variable of behavioral intentions to use services that are em-
bedded in UIS. Hereby, also discriminant validity of SSF with regard to perceived usefulness and per-
ceived ease of use is assessed. For this purpose, data from a four-year EU project is used, which in-
cludes a pretest and three empirical user studies. Results show that SSF can be a significant predictor
of service usage. However, validity issues still exist and thus, further research on the SSF instrument
is recommended.
Keywords: Ubiquitous Information System, Design Method, Situation-Service Fit, Empirical Study,
Instrument evaluation.
1 Introduction
One of the core research activities of the Information Systems (IS) community during the last forty
years is the investigation into the “effective design, delivery, use and impact of information technolo-
gy in organizations and society.” (Avison and Fitzgerald, 1995, p. xi) Accordingly, the core objective
of IS research is to increase the efficiency of business organizations (Hevner et al., 2004). In contrast
to the more traditional IS in the organizational context and during the last two decades, however, in-
formation and communication services have become ubiquitous in our everyday life: “always-on” In-
ternet-connected mobile phone for electronic commerce activities (Frolick and Lei-da-Chen, 2004;
Mennecke and Strader, 2002), a radio-frequency enhanced lift ticket (O'Connor, 2008) or a Wi-Fi ena-
bled refrigerator with a digital shopping list and calendar functionality
1
. These kinds of applications
have been discussed under umbrella terms such as ambient intelligence (Keegan et al., 2008), experi-
ential computing (Yoo, 2010), internet of things (Fleisch and Mattern, 2005; Floerkemeier et al.,
2008), nomadic computing (Lyytinen and Yoo, 2002b), pervasive computing (Hansmann et al., 2012;
Orwat et al., 2008), smart products (Thiesse and Köhler, 2008), ubiquitous computing (Cousins and
Varshney, 2009; Lyytinen and Yoo, 2002a) or ubiquitous information systems (Vodanovich et al.,
2010). All of these terms can be traced back to the seminal article entitled “The computer for the 21st
Century” by Mark Weiser (Weiser, 1991). Since the current work is written predominantly for the IS
community, the term ubiquitous information system (UIS) is adopted. In line with prior work, UIS is
defined as a composite of people, processes, information and communication services embedded in
hardware that is interconnected and interwoven into our lives (Vodanovich et al., 2010).
Computer scientists have successfully set up rather technical conferences on UIS such as the IEEE
PerCom or ACM UbiComp. A few prominent topics at these conferences are devices and infrastruc-
tures, adaptive, autonomic and context-aware computing systems or the use of radio-frequency identi-
fication in pervasive systems. IS researchers, on the other hand, are rather interested in the socio-
technical aspects of information systems. That is, they focus predominantly on the impact of technolo-
gy on organizations or individuals. However, the IS community has rather slowly adopted research on
UIS over the last two decades even though research opportunities exist to a great extent (Yoo, 2010).
For example, only few articles and research commentaries can be found in top IS journals (e.g.
Lyytinen and Yoo, 2002b; Vodanovich et al., 2010; Yoo, 2010).
Questions arise therefore whether UIS are distinct from traditional IS in organizations and, if so,
whether they demand investigation with native research instruments. We will address both questions
with the following four dimensions proposed by Vodanovich et al. (2010): (1) Context: work versus
home, (2) Activity: professional versus personal, (3) User: digital immigrant versus digital native and
(4) Technology: traditional versus ubiquitous. First, UIS are not restricted to traditional (business) or-
ganizations anymore but consider individuals or groups being at home, in the shopping mall, with
friends at a party or on vacation. The context of UIS use is therefore extended from the traditional of-
fice environment to non-office and home environments. Second, activities supported by UIS include
not only professional tasks such as managing customer requests, issuing invoices or writing business
emails but also personal activities like meeting friends or personal shopping. Third, the user has
changed over the last two decades. Digital natives, which are people that grew up with digital technol-
ogy from the very first days of their lives, have adopted different behavioral patterns while using
ubiquitous technologies than those digital immigrants that were born a generation before (Prensky,
2001). For example, using Facebook or Foursquare
2
on mobile devices at any time and in any place is
perceived as natural for digital natives whereas at least digital immigrants without a strong technical
background are required to change their traditional approach towards office computing or personal
1
http://www.samsung.com/us/appliances/refrigerators/RF4289HARS/XAA
2
https://foursquare.com/about/new
computing if they want to benefit from UIS. Finally, ubiquitous technologies are described as “hard-
ware such as tabs, pads, boards, dust, skins, and clay that are interconnected and interwoven into the
very fabric of our lives through ubiquitous networks.“ (Vodanovich et al., 2010, p. 713) By contrast,
system consists of “people, processes, information, and communication systems and technologies.”
(ibid.) UIS are therefore systems that embed ubiquitous technologies to support individuals or groups
whereas computer systems such as desktop computers require users to actively approach and sit down
at those kinds of traditional systems. UIS differ also in form, size and appearance from traditional sys-
tems such as desktop computers.
In summary, there is a fundamental need to better understand the design, deployment, use and impact
of UIS. As a first step, this work focuses on the design of empirically validated UIS and is guided by
the general research question: How can UIS be evaluated rigorously? This question will be answered
with findings from an empirical instrument evaluation test and three evaluations of one particular UIS.
The remainder of this work is structured as follows. Next, related work on a design methodology tai-
lored to UIS as well as existing IS evaluation instruments are provided to frame the current work.
Then, particular research questions are stated and it is outlined by which methodological approach
they are addressed. Afterwards, the research questions are answered by discussing how UIS design
artifacts of different levels of abstraction can be evaluated empirically. In particular, findings from a
pretest and three empirical studies are reported and interpreted. Implications of these findings and lim-
itations of the current work are then discussed from which an outlook on future research is derived.
2 Related Work
2.1 Situational Design Method for IS (SiDIS)
Designing UIS requires a common understanding of the involved stakeholders. For that purpose, mod-
els can be used which are abstractions of the real world (Maxim, 1999). They help to better understand
what a particular UIS is all about, i.e. how it works, what information it requires and processes, which
persons are supported by it and finally, which services are embedded into which sort of physical ob-
jects (e.g. tablet PC, furniture or other kinds of everyday objects). Depending on the level of abstrac-
tion, UIS can be modeled by narratives (i.e. textual descriptions), conceptual models, mock-ups or
prototypes, all of which can be empirically evaluated at different kinds of stages of development.
In order to design UIS, the transitions from high-level abstractions (e.g. narratives) to more concrete
forms of UIS representations (e.g. mock-ups or prototypes) should rely on justified design decisions.
For this purpose, several methodologies currently exist, in particular from the engineering and com-
puter science disciplines that guide the design process of information systems. Those methodologies
usually include several steps such as problem definition, requirements engineering, formal system de-
sign, technical architecture, prototyping, implementation, testing, deployment and maintenance. Ex-
amples are the waterfall model or the rational unified process (Larman and Basili, 2003).
With regard to UIS, twelve design methodologies for IS according to relevant design principles were
reviewed and, due to a lack of coverage of these principles, the Situational Design Method for IS
(SiDIS) was proposed (Janzen et al., 2010; Maass and Janzen, 2011). SiDIS consists of four phases:
(1) identification of the problem and needs, (2) design of the solution, (3) development of the solution
and (4) evaluation of the solution. At its core, this methodology guides the design process from a high-
ly level of abstraction of UIS representations to more concrete representations as described above. It
also requires developers to build and continuously evaluate UIS representations. Thus, build and eval-
uate loops as proposed in design science research (Hevner et al., 2004; Peffers et al., 2007) are an in-
tegral part of each phase of SiDIS. That is, the primary objective of the design activities in each phase
is the development of empirically justified artifacts. Examples are narratives, diagrammatic models,
virtual or real mock-ups, prototypes or, UIS solutions that are ready to be deployed in the market. An
overview of SiDIS is presented in Figure 1. In order to better understand the various design phases in
which UIS can be evaluated, the current work refers to SiDIS to address its research questions.
Figure 1. Overview of the Situational Design Method for IS (SiDIS)
2.2 IS Evaluation Instruments
The IS community traditionally performs empirical studies in the context of (business) organizations.
One of the research questions is usually: What are the success factors of IT use? Two streams of re-
search have primarily influenced these empirical studies. First, research on diffusion of innovations
(Rogers, 2003) takes a social science perspective into account and argues that perceived characteristics
of innovations, e.g. relative advantage, trialability, observability or compatibility, predict their adop-
tion (Moore and Benbasat, 1991). Second, theory of reasoned action (Fishbein and Ajzen, 1975) and
its successor, the theory of planned behavior (Ajzen, 1991), have been adopted by IS researchers from
the field of social psychology. These theories describe attitudes, beliefs and perceived constraints that
influence the behavior of individuals. In particular, the behavioral intention to use IT and actual (con-
tinuous) IT use are here the dependent variables in question.
Based upon these research streams, empirical models have been proposed and tested. Examples are the
Unified Theory of Acceptance and Use of Technology (UTAUT) (Venkatesh et al., 2003), UTAUT2
(Venkatesh et al., 2012) or the (Updated) DeLone and McLean models of IS success (DeLone and
McLean, 1992; DeLone and McLean, 2003). One of the more widely used empirical models is the
Technology Acceptance Model (TAM) (Davis, 1989). It states that perceived usefulness and perceived
ease of use predict IT use through behavioral usage intention as mediating factor. TAM was developed
in the organizational context and thus, scale items include terms like useful in my job or my effective-
ness on the job. That is, TAM is used to evaluate IS artifacts that support well-defined tasks.
By contrast, it is argued that UIS-specific constructs are required to capture dedicated situational char-
acteristics that are often unpredictable and not well defined (Maass and Janzen, 2011; Maass et al.,
2012). Accordingly, Maass et al. (2012) proposed the Situation-Service Fit (SSF) item for the evalua-
tion of UIS, which is defined as “a perceived degree of match between a service and a situation.” The
following three dimensions complement the definition of this overall SSF item:
• Behavior-Service Fit is the degree to which an individual believes that a particular service is com-
patible with his or her own behavior.
• Modality-Service Fit is the degree to which a person believes that a service supports him or her in a
feasible way of providing information (e.g. in the form of text, audio or video) and interacting with
information (e.g. text, gestures or speech) according to a given situation.
1. Identification of
the Problem
and Needs
2. Design of
the Solution
3. Development of
the Solution
4. Evaluation of the
Solution
Activity Artifact
Build &
Evaluate
Build &
Evaluate
Build &
Evaluate
Build &
Evaluate
Problem descriptions
Evaluation
Lab studies
Field studies
Engineering
test methods
Workshops
Phase
Prototype / Solution
Mock-up
I get weather
information ...
Narrative
Lab studies
Propositional conceptual models, technical artifact
• Spatial-Service Fit is the degree to which a person believes that the point of presentation or interac-
tion with a particular service fits a particular situation.
3 Methodology
Up till now, existing work of the co-authors (Maass et al., 2012) has only reported descriptive statis-
tics and one-sample t-tests of the SSF items and thus, it is still open how UIS can be evaluated rigor-
ously. In particular, it is relevant to know (1) whether SSF shows construct validity, convergent validi-
ty and discriminant validity, (2) how SSF is used within the SiDIS design process and (3) to which
degree SSF influences intentions to use UIS services. The rest of this section therefore aims at answer-
ing these questions as follows.
1. Testing construct validity, discriminant and convergent validity of the SSF instrument
a. by a pretest in section 3.1 and
b. by factor analyses with data from two empirical studies in sections 3.3 and 3.4.
2. Showing how SSF is applied in SiDIS by three empirical studies in sections 3.2, 3.3 and 3.4.
3. Using multiple regression analyses to test whether SSF influences the intention to use one par-
ticular UIS service with data from two empirical studies in sections 3.3 and 3.4.
3.1 Pretest: Item sorting
The objective of this pretest is to ensure construct validity, convergent validity and discriminant va-
lidity of the SSF construct. Consistent with prior work (Moore and Benbasat, 1991), an item sorting
task was performed. For that purpose, four SSF items, four perceived usefulness items and four per-
ceived ease of use items were used (cf. appendix). Two doctoral students, two research assistants and
two master students from an IS department of a European University were asked to sort the twelve
items into two to five categories. After they had finished this sorting round, they had to give their cat-
egories a concrete name, too. Overall, this pretest was done individually by each participant and took
twenty minutes on average.
As a result of this item sorting task, participants have named the categories as follows: raise of effi-
ciency, usefulness, utility and effects on me for the perceived usefulness items; understandability, ease
of use, handling and usability for the perceived ease of use items; and finally, adaptation, fit, service
and situation, correct situation and contextualization for the SSF items. Because the proposed names
show face validity as they represent various dimensions of the theoretical constructs in question, it can
be assumed that the new SSF items support construct validity.
A further indicator of construct validity can be obtained from the item placement ratios as shown in
Table 1. These ratios were calculated from the sorting task and are used to demonstrate convergent
validity with the underlying theoretical construct and discriminant validity with the other constructs.
Results indicate that the overall hit ratio lies by over 95 percent. That is, most items have been placed
into the theoretical correct category as shown by the diagonal values. Only one time, an SSF item was
assigned to the perceived usefulness construct which shows excellent convergent and discriminant va-
lidity.
In order to test inter-rater reliability, the raw agreement was used as well as the more rigorous Cohen’s
Kappa (Cohen, 1960), which also considers the likelihood that subjects assign items into correct cate-
gories incidentally. Both measurements range between zero (no agreement) and one (full agreement).
The average agreement scores for all 15 pairs resulted in viable .92 and a standard deviation of .08 for
the raw agreement and remarkable .88 and a standard deviation of .13 for Cohen’s Kappa.
In summary, results show good reliability such that construct validity, convergent validity and discri-
minant validity of the SSF construct are assumed.
Actual
Theoretical
PU
PEU
SSF
N/A
Total
% Hits
Perceived Usefulness (PU)
23
0
1
0
24
95.8
Perceived Ease of Use (PEU)
1
23
0
0
24
95.8
Situation-Service Fit (SSF)
0
1
23
0
24
95.8
Total Item Placements
72
Hits (diagonal values)
69
Table 1. Item placement ratios
3.2 Study 1: Evaluation of a UIS Narrative
The first empirical evaluation was conducted in the second phase of SiDIS, i.e. design of the solution.
Accordingly, problem descriptions have been already identified in the first phase of SiDIS and were
now transformed into narratives. Narratives are UIS representations in form of textual descriptions of
everyday situations. An example would be the following narrative that describes the weather infor-
mation service as part of our case study, too:
Anna gets site-specific weather information when she is brushing her teeth in the bathroom.
Based on weather information and her calendar, free-time event suggestions are given, e.g. To-
day, 8 p.m. — Sneak Preview at CinemaOne. Do you want to order tickets? (Janzen et al., 2010)
By rating twelve preliminary narratives, Study 1 had the objective to identify the most relevant of
them for further development. This study was also the first that used an early version of an SSF item.
The item wording was the bathroom situation would fit very well into my private life. Overall, 46 sub-
jects from Germany (N=22, students and university staff from two computer science departments),
Switzerland (N=13, doctoral students of an IS department) and Turkey (N=11, students from a com-
puter science department) evaluated each narrative by indicating their consent with that item on a sev-
en-point Likert scale ranging from strongly disagree (1) to strongly agree (7). The SSF item was used
exclusively in this study rendering the situation as such and, implicitly, the individual behavior of the
user as the only evaluation criteria.
However, subjects were not able to appropriately evaluate narratives with this item. That is, narratives,
like the one presented above, might contain several services that subjects evaluate differently. In the
worst case, the perceived degree of fit of service A would be very high whereas perceived fit of ser-
vice B would be the opposite. As a result, subjects are indifferent rendering narratives neither positive
nor negative. At the end, service A would have no chance to be considered for further development.
Based on this observation, the following three guidelines are given for future evaluation of narratives:
• Decompose narratives: A narrative can be decomposed into a single-service narrative and evalu-
ated with the SSF item. The drawback would be, however, to lose the stickiness of more complex
situations we face in our everyday life.
• Evaluate each service as such relative to the whole situation: The SSF items of the pretest are
appropriately designed such that narratives are not holistically evaluated. The drawback, however,
is that each service embedded in a narrative would require an individual item. Thus, this approach
results in an overall longer evaluation instrument that researchers have to cope with.
3
3
Nevertheless, this guideline was adopted for the mock-up-based evaluation in Study 2 and the prototype-based evaluation in
Study 3 (cf. sections 3.3 and 3.4).
• Systematically manipulate service configurations: First, narratives should be created with differ-
ent services configurations. Then, conjoint analysis can help to identify the most relevant service
configuration by comparisons. For the comparison procedure, SSF statements can be used. The
drawback here is that the planning phase of the evaluation is more complex.
As a general guideline at this stage of UIS designs, it is recommended that Situation-Service Fit and
Situation-Behavior Fit of SSF are used. That is, because narratives do explicitly not provide any cues
on technical details which subjects would require to appropriately rate services with the remaining two
SSF dimensions Modality-Service Fit and Spatial-Service Fit.
In summary, SSF can be used to evaluate and identify relevant services that are embedded in everyday
situations based upon a pool of preliminary narratives. However, researchers should consider the
guidelines and drawbacks described above.
3.3 Study 2: Evaluation of a UIS Mock-up
The second empirical evaluation was also part of the second phase of SiDIS. But in contrast to Study
1, subjects had to evaluate only the top rated bathroom situations from Study 1. And compared to nar-
ratives, a mock-up including supplementary slides on a laptop represented the UIS now. The mock-up
was a paper-based miniature of a bathroom with dolls whereas more detailed information about the
services was provided by the slides (cf. Figure 1).
The objective of this sort of study is to add validity to the findings of Study 1 and to further reduce the
amount of relevant bathroom situations in order to handle the rather expensive third phase of SiDIS,
i.e. development of the solution. Various empirical constructs such as perceived usefulness (PU) or the
intention to use a particular service were used for that purpose. Seven-point Likert scales from strong-
ly disagree (1) to strongly agree (7) were used. This study was also the first that employed Behavior-
Service Fit (BSF) and thus, compared to Study 1, narratives as such were not target of investigation
anymore. Overall, 111 subjects were recruited from the same universities as for Study 1, i.e. they came
from Germany (N=63, students and university staff from two computer science departments), Switzer-
land (N=36, doctoral students of an IS department) and Turkey (N=12, students from a computer sci-
ence department).
Regarding the current work, data of the weather information service from Study 2 is now used to test
convergent and discriminant validity of BSF. Because perceived ease of use was excluded from the
study due to the non-functional mock-up-based representation of situations, only three PU items
adapted from (Wixom and Todd, 2005) were used for this test. Consistent with prior work (Moore and
Benbasat, 1991), confirmatory factor analysis with principal components extraction and varimax rota-
tion was performed. With a value of .644, the Kaiser-Meyer-Olkin criterion indicates a mediocre
measure of sampling adequacy (Hutcheson and Sofroniou, 1999). The first factor with an eigenvalue
of 2.6, interpreted as PU, explains 65.3% of the variance whereas the second factor, interpreted as
BSF, adds 22.9% of explained variance. However, with an eigenvalue of .917, the second factor lies
slightly below the Kaiser criterion of one, a cut-off criterion which often overestimates the number of
factors (Lance et al., 2006). Consistently, parallel analysis (Horn, 1965) confirms a one-factor solu-
tion.
4
In addition, the PU3 item loads clearly on the second factor as shown in Table 2. This is con-
firmed by reliability analysis, as Cronbach’s Alpha (Nunnally, 1967) would increase from .82 to .91 if
PU3 is dropped. It is assumed that there exists a problem with regard to the convergent validity of the
PU scale and thus, PU3 is dropped from further analysis. After elimination, convergent and discrimi-
nant validity of the PU items and SBF items can be assumed by interpreting the values after the slash
of Table 2 and taking an artificial two-factor solution for granted at this preliminary stage of research.
4
https://people.ok.ubc.ca/brioconn/nfactors/rawpar.sps was used with SPSS 20.0.0.1 and raw data permutation.
With the two factor values from the second factor analysis, it was then evaluated to which degree BSF
influences the behavioral intention to use the weather information service relative to PU. Multiple re-
gression analysis was used for that purpose. Results indicate that with a score of .519 adjusted R
Square is significant at the .001 level. Both variance inflation factors (VIFs) (Backhaus et al., 2006)
are one indicating that multi-collinearity can be neglected. Furthermore, both factors have a significant
relationship with the dependent variable at the .001 level. However, standardized beta coefficients
show that the BSF factor is quite more relevant with .689 than the PU factor with a score of .235. All
in all, it can be assumed in the context of this study and within the logic of TAM that the effect of SBF
on behavioral intentions of service usage is by far more relevant than the effect of PU.
Item
Factor 1
Factor 2
PU1: Using [service] would enhance my effectiveness*
.92 / .94
.26 / .20
PU2: Using [service] would enhance my efficiency*
.93 / .94
.21 / .17
PU3: Using [service] would be useful for me*
.33 / —
.85 / —
BSF: Using [service] would fit very well to my behavior*
.14 / .19
.92 / .98
Table 2. Rotated factor matrix before / after dropping PU3 (N=111). Note: * in the bathroom
3.4 Study 3: Evaluation of a UIS Prototype
The third empirical evaluation was part of the fourth phase of SiDIS, i.e. evaluation of the solution. In
this study, situations and services from Study 2 have remained the same due to the fact that particular
services were top rated throughout all situations. But in contrast to Study 2, the situations were now
evaluated based on a prototype that has been developed during the third phase of SiDIS (cf. Figure 2).
That is, an interactive service-enhanced bathroom represented the UIS. The objective of this study was
therefore to add external validity to the findings from Study 2 and to capture detailed requirements for
a refined UIS version. This study was also the first in which all four SSF dimensions were used to-
gether with the TAM constructs. In detail, Modality-Service Fit (MSF) and Spatial-Service Fit (SPSF)
were used in addition to Behavior-Service Fit (BSF) and Situation-Service Fit (SSF), because services
were implemented physically and subjects were now able to rate these dimensions. Because the proto-
type of the interactive bathroom was only built up in Germany only, 27 students from a computer sci-
ence department of a German University) evaluated the weather information service.
To be consistent with Study 2 while testing discriminant validity and convergent validity, confirmato-
ry factor analysis was used with principal components extraction and varimax rotation with the PU
items and the SSF, BSF, MSF and SPSF items as shown in Table 3. But now the analysis resulted in a
three-factor solution based on the Kaiser criterion, i.e. with eigenvalues greater than one.
5
The Kaiser-
Meyer-Olkin measure of sampling adequacy yielded a middling value of .729. The three factors ex-
plained 42% (interpreted as PU), 16% (interpreted as composite of MSF and SPSF) and 14% (inter-
preted as SSF) of the variance. Moreover, BSF loads clearly on the PU factor. It thus becomes appar-
ent that the SSF, BSF, MSF and SPSF items do not show convergent validity and discriminant validity
based on this data. Because only SSF, MSF and SPSF are distinct from PU, BSF was dropped for fur-
ther analysis. The resulting factor loadings are shown in Table 3 after the slash.
Then the effects of PU and the remaining three SSF items on the behavioral intention to use the
weather information service were tested with multiple regression analysis. For that purpose, PU items
were aggregated because of their high internal consistency (Alpha = .84) whereas the SSF, MSF and
SPSF items were taken separately into account because of the results of factor analysis and their low
5
It must be noted that a comparison with the results from parallel analysis (cf. section 3.3 for further details) revealed a one-
factor solution as in Study 2. Thus, the results must be interpreted carefully.
internal consistency (Alpha = .29). Adjusted R Square is significant at the .001 level and yields a rela-
tively high value of .634. All VIFs lie between 1.0 and 1.1 indicating that multi-collinearity can be
neglected. With .721 the standardized beta coefficient of PU is significant at the .001 level. Only the
coefficient of the SSF item is also significant at the .05 level with a value of .305. This shows surpris-
ingly and in contrast to the results of Study 2 that PU explains more variance on behavioral usage in-
tentions than the SSF item.
Construct / Item
Factor 1
Factor 2
Factor 3
Perceived Usefulness (PU)
PU1: Using [service] * would increase my effectiveness.
.79 / .79
-.25 / -.31
.16 / .21
PU2: Using [service] * would increase my efficiency.
.92. / 93
.13 / .08
-.14 /-.11
PU3: Using [service] * would increase my overall performance.
.72 / .77
.34 / .26
-.09 /-.11
PU4: I would find the [service] useful *.
.80 / .82
.17 / .13
.07 / .09
Situation-Service Fit
SSF: [service] fits well to the situation I just played through.
.11 / .10
.06 / .06
.94 / .92
BSF: Using [service] * would fit well to my behavior.
.73 / —
-. 07 / —
.32 / —
MSF: The modality of content presentation fits well to [service].
.35 / .34
.65 / .68
-.21 / -.30
SPSF: The spatial placement of the contents fits well to [service].
-.15 /-.09
.84 / .81
.25 / .31
Table 3. Rotated factor matrix before / after dropping BSF. Note: * in the bathroom
4 Implications, Limitations and Future Research
Table 4 depicts the overall results with regard to the research questions stated in section 3. Results in-
dicate that the application of SiDIS and the empirical data of the pretest and the three studies are gen-
erally helpful for answering the research questions. However, with regard to the results addressing the
first and third question, current findings must be perceived as preliminary because they are not com-
pletely consistent with each other. Three reasons might be relevant for the interpretation of these in-
consistencies and the discussion of corresponding implications.
First of all, individual subjects of the pretest and the three studies were not held constant. Thus, the
participants differ in age, educational and cultural background. This is also one of the primary limita-
tions of the current work. With this explanation, the inconsistencies with regard to the first research
question (pretest versus Study 2 and Study 3) and the third research question (Study 2 versus Study 3)
can be probably explained.
Second, validity issues might be further reasons for the inconsistencies of the results. Although the
pretest has clearly indicated that SSF shows construct validity (being aware that participants of this
pretest were rather semi-professionals in the IS field), convergent validity and discriminant validity
when compared to perceived usefulness and perceived ease of use, the factor analyses of Study 2 and
Study 3 have revealed that (1) eigenvalues of the extracted factors must be interpreted carefully, (2)
BSF as one dimension of the SSF construct overlaps with perceived usefulness and (3) MSF loads on
the same factor as SPSF. Whereas the former result indicates that BSF might be a relevant predictor of
perceived usefulness (Benbasat and Barki, 2007), the latter result indicates the existence of a subgroup
of the SSF construct, which could be termed as interaction-service fit because these items evaluate the
modality, i.e. how to interact with UIS services, and the spatial placement, i.e. where to interact with
UIS services.
And a third reason for the inconsistencies might be the different levels of abstractions of the UIS rep-
resentations. That is, the narratives used in Study 1 are more abstract and focus rather on the quintes-
sence of a UIS compared to Study 3, in which the participants could actually experience the UIS and
interact with it. As a result, it is assumed that different conceptual models of the UIS were constructed
in the mind of the subjects, which had, in turn, an influence on their evaluations. However, it would be
interesting to assess the predicting value of relevant constructs of mock-ups compared to prototypes of
UIS, which addresses in particular the third research question. Given a comparable sample of subjects
and comparable correlations between the SSF construct and behavioral intentions to use UIS services
for both mock-up-based and prototype-based evaluations, the design of UIS could be conducted under
rather low-budget conditions and would therefore foster the design of UIS in general. That is, mock-
ups and their evaluations would have a high predictive value or, in other words, would be cost-
efficient and high approximations of the real UIS. Future research should therefore address this re-
search topic in the context of UIS, also because it as it has been already done for more traditional in-
formation systems in the past (e.g. Davis and Venkatesh, 2004).
Data from
Research Question f ee
Pretest
(N=6)
Study 1: Evalua-
tion of narratives
(N=46)
Study 2: Evaluation
of a mock-up
(N=111)
Study 3: Evalua-
tion of a UIS
prototype(N=27)
1.
Is Situation-Service Fit (SSF) a
valid construct?
✔
Item sorting
n/a
(✔)
Factor analysis – but: overlapping with
perceived usefulness
2.
How can SSF be applied in the
SiDIS design process?
n/a
✔ Identification and evaluation of relevant services and
situations for the design of UIS
3.
Is SSF a relevant construct for
the adoption of UIS?
n/a
n/a
✔
Regression
analysis
(✔)
Regression
analysis
Table 4. Research questions and results of the current work.
Note:
✔
=yes, (
✔
) =preliminary yes further / future research
In order to crosscheck the inconsistencies described above, the current analyses should be repeated
with further UIS services. Empirical data of subjects that participated from different European coun-
tries should be also individually analyzed to account for any cultural bias. Special consideration must
also be given to predicting factors of behavioral usage intentions. Based on the results of Study 2 and
3 it is assumed that subjects’ evaluations and preferences may change with the target of investigation,
be it a mock-up or a prototype. Thus, further empirical studies and analyses are required first. If the
findings are then still inconsistent, a rigorous instrument development process should be performed for
the four SSF dimensions. Prior work can guide this process (Davis, 1989; Moore and Benbasat, 1991).
However, research on UIS probably requires an adaptation of the development methods for evaluation
instruments. That is, the complexity of everyday situations and UIS services that support not only in-
dividuals but also groups should be considered as well.
All in all, findings from the Pretest as well as Study 1 and Study 2 are promising in the sense that UIS
require dedicated instruments that might overrule traditional predictors of IS use such as those ante-
cedents of TAM. In line with all the applications introduced at the beginning of this paper, it is there-
fore strongly recommended to investigate further into the individual characteristics of UIS.
References
Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision
Processes, 50 (2), 179-211.
Avison, D. and Fitzgerald, G. (1995). Information Systems Development: Methodologies, Techniques
and Tools. (2nd ed.) McGraw-Hill, London, UK.
Backhaus, K., Erichson, B., Weiber, W. and Plinke, R. (2006). Multivariate Analysemethoden. Eine
anwendungsorientierte Einführung. (11th ed.) Springer, Berlin.
Benbasat, I. and Barki, H. (2007). Quo vadis, TAM? Journal of the Association for Information
Systems, 8 (4), 211-218.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological
Measurement, 20, 37–46.
Cousins, K.C. and Varshney, U. (2009). Designing Ubiquitous Computing Environments to Support
Work Life Balance. Communications of the ACM, 52 (5), 117-123.
Davis, F.D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information
Technology. MIS Quarterly, 13 (3), 319-339.
Davis, F.D. and Venkatesh, V. (2004). Toward preprototype user acceptance testing of new
information systems: Implications for software project management. IEEE Transactions on
Engineering Management, 51 (1), 31-46.
DeLone, W.H. and McLean, E.R. (1992). Information systems success: The quest for the dependent
variable. Information Systems Research, 3 (1), 60-95.
DeLone, W.H. and McLean, E.R. (2003). The DeLone and Mc Lean Model of Information Systems
Success: A Ten-Year Update. Journal of Management Information Systems, 19 (4), 9-30.
Fishbein, M. and Ajzen, I. (1975). Belief, Atitude, Intention and Behaviour: An Introduction to Theory
and Research Addison-Wesley, Reading, MA.
Fleisch, E. and Mattern, F. (2005). Das Internet der Dinge Springer, Berlin.
Floerkemeier, C., Langheinrich, M., Fleisch, E., Mattern, F. and Sarma, S.E. (2008). The Internet of
Things - First International Conference, IOT 2008, Zurich, Switzerland, March 26-28, 2008,
Proceedings Springer, Heidelberg, Germany.
Frolick, M.N. and Lei-da-Chen, M.N. (2004). ASSESSING M-COMMERCE OPPORTUNITIES.
Information Systems Management, 21 (2), 53-61.
Hansmann, U., Merk, L., Nicklous, M.S. and Stober, T. (2012). Pervasive Computing Springer,
Heidelberg, Germany.
Hevner, A.R., March, S.T., Park, J. and Ram, S. (2004). Design Science in Information Systems
Research. MIS Quarterly, 28 (1), 75-105.
Horn, J.L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrica, 30,
179-185.
Hutcheson, G. and Sofroniou (1999). The Mulitvariate Scientist Sage, London.
Janzen, S., Kowatsch, T. and Maass, W. "A Methodology for Content-Centered Design of Ambient
Environments," in: Global Perspectives on Design Science Research, 5th International Conference,
DESRIST 2010, St. Gallen, Switzerland, June 4-5, 2010 Proceedings, R. Winter, J.L. Zhao and S.
Aier (eds.), Springer, Berlin, Germany, 2010, pp. 210-225.
Keegan, S., O'hare, G.M.P. and O'grady, M.J. (2008). EasiShop: Ambient Intelligence Assists
Everyday Shopping. Journal of Information Sciences, 178 (3), 588-611.
Lance, C.E., Butts, M.M. and Michels, L.C. (2006). The Sources of Four Commonly Reported Cutoff
Criteria: What Did They Really Say? Organizational Research Methods, 9 (2), 202-200.
Larman, C. and Basili, V.R. (2003). Iterative and Incremental Development: A Brief History.
Computer, 36 (6), 47-56.
Lyytinen, K. and Yoo, Y. (2002a). Issues and Challenges in Ubiquitous Computing. Communications
of the ACM, 45 (12), 62-65.
Lyytinen, K. and Yoo, Y. (2002b). Research Commentary: The Next Wave of Nomadic Computing.
Information Systems Research, 13 (4), 377-388.
Maass, W. and Janzen, S. "Pattern-Based Approach for Designing with Diagrammatic and
Propositional Conceptual Models," in: Service-oriented Perspectives in Design Science Research,
6th International Conference, DESRIST 2011, Milwaukee, WI, USA, May 5-6, 2011, H. Jain, A.P.
Sinha and P. Vitharana (eds.), Springer, Heidelberg, Germany, 2011, pp. 192-206.
Maass, W., Kowatsch, T., Janzen, S. and Filler, A. (2012). Applying Situation-Service Fit to Physical
Environments Enhanced by Ubiquitious Information Systems. In Proceedings of the 20th European
Conference on Information Systems (ECIS), Barcelona, Spain.
Maxim, P.S. (1999). Quantitative Research Methods in the Social Sciences Oxford University Press,
New York.
Mennecke, B. and Strader, T. (eds.) Mobile Commerce: Technology, Theory and Applications. Idea-
Group Publishing, Hershey, PA, 2002.
Moore, G.C. and Benbasat, I. (1991). Development of an instrument to measure the perceptions of
adopting an information technology innovation. Information Systems Research, 2 (3), 192-222.
Nunnally, J.C. (1967). Psychometric Theory McGraw-Hill, New York.
O'Connor, C. (2008) Alta Opts for RFID Lift Tickets. http://www.rfidjournal.com/article/print/4110
Orwat, C., Graefe, A. and Faulwasser, T. (2008). Towards pervasive computing in health care – A
literature review. BMC Medical Informatics and Decision Making, 8 (26), 19.
Peffers, K., Tuunanen, T., Rothenberger, M. and Chatterjee, S. (2007). A Design Science Research
Methodology for Information Systems Research. Journal of Management Information Systems, 24
(3), 45-77.
Prensky, M. (2001). Digital Natives, Digital Immigrants: a New Way To Look At Ourselves and Our
Kids. On the Horizon, 9 (5), 2-6.
Rogers, E.M. (2003). Diffusion of innovations. (5 ed.) Free Press, New York.
Thiesse, F. and Köhler, M. (2008). An Analysis of Usage-Based Pricing Policies for Smart Products.
Electronic Markets, 18 (3), 232-241.
Venkatesh, V., Morris, M.G., Davis, G.B. and Davis, F.D. (2003). User acceptance of information
technology: Toward a unified view. MIS Quarterly, 27 (3), 425-478.
Venkatesh, V., Thong, J.Y.L. and Xu, X. (2012). Consumer Acceptance and Use of Information
Technology: Extending the Unified Theory of Acceptance and Use of Technology. MIS Quarterly,
36 (1), 157-178.
Vodanovich, S., Sundaram, D. and Myers, M. (2010). Digital Natives and Ubiquitous Information
Systems. Information Systems Research, 21 (4), 711-723.
Weiser, M. (1991). The computer for the 21st century. Scientific Amercian, 265 (3), 94-104.
Wixom, B.H. and Todd, P.A. (2005). A Theoretical Integration of User Satisfaction and Technology
Acceptance. Information Systems Research, 16 (1), 85-102.
Yoo, Y. (2010). Computing in Everyday Life: A Call for Research on Experiential Computing. MIS
Quarterly, 34 (2), 213-231.
Appendix – Items of the Pretest
Perceived Usefulness (PU) Items were adapted from Davis (1989) and Wixom and Todd (2005).
• PU1: Using this service in the bathroom increases my effectiveness.
• PU2: Using this service in the bathroom increases my efficiency.
• PU3: Using this service in the bathroom increases my overall performance.
• PU4: I find this service useful in the bathroom.
•
Perceived ease of use (PEU) Items were adapted from Davis (1989) and Wixom and Todd (2005).
• PEU1: Using this service is easy for me.
• PEU2: My interaction with this service is clear and understandable.
• PEU3: Learning to use this service is easy for me.
• PEU4: It was easy for me to become skillful at using this service.
•
Situation-Service Fit (SSF) Items were adapted from Maass et al. (2012).
• Situation-Service Fit: This service fits well to the situation.
• Behavior-Service Fit: Using this service in the bathroom fits well to my behavior.
• Modality-Service Fit: I find that the modality of content presentation fits well to the service.
• Spatial-Service Fit: I find that the spatial placement of the contents fits well to this service.
