MethodPDF Available

Subjective Information Processing Awareness Scale (SIPAS)

Authors:
Tim Schrills at University of Lübeck
Tim Schrills
Thomas Franke at University of Lübeck
Thomas Franke
Download file PDFDownload file PDF
Read file
Download citation

Abstract

SIPA can be defined as the experience of being enabled by a system to perceive, understand, and predict its information processing. The SIPA scale is a highly economical unidimensional scale focused on an application in Explainable Artificial Intelligence, as was shown in first empirical studies (Schrills et al., 2021).
SIPAS_eng_09.20
21.pdf
Content uploaded by Tim Schrills
Author content
All content in this area was uploaded by Tim Schrills on Nov 29, 2021
Content may be subject to copyright.
SIPAS_deu_09.20
21.pdf
Content uploaded by Tim Schrills
Author content
All content in this area was uploaded by Tim Schrills on Nov 29, 2021
Content may be subject to copyright.
Subjective Information Processing Awareness Scale (SIPAS)
Subjective Information Processing Awareness (SIPA) describes the experience of users during their
interaction with intelligent systems and is grounded in Situation Awareness theory (Endsley, 1988).
SIPA can be defined as the experience of being enabled by a system to perceive, understand, and
predict its information processing.
The SIPA scale is a highly economical unidimensional scale focused on an application in Explainable
Artificial Intelligence, as was shown in first empirical studies (Schrills et al., 2021)
The following questionnaire deals with your experience in the interaction with the system.
Information refers to all data that the system can work with. Result refers to the output of the
system, which is presented at the end of the system's information processing.
Please indicate the degree to which you agree/disagree
with the following statements.
completely
disagree
largely
disagree
slightly
disagree
sligthly
agree
largely
agree
completely
agree
01
It was transparent to me which information was
collected by the system.
02
The information that the system could acquire
was observable for me.
03
It was understandable to me how the collected
information led to the result.
04
The system's information processing was
comprehensible to me.
05
With the information accessible for me, the result
was foreseeable for me.
06
The system's information processing was
predictable for me.
Analysis:
When entering the participants’ responses into a data file for the analysis, the responses should be
coded as follows: completely disagree = 1, largely disagree = 2, slightly disagree = 3, slightly agree =
4, largely agree = 5, completely agree = 6. A mean score should be computed over all 6 items.
When to apply:
The SIPA scale can be applied in contexts where humans cooperate with intelligent systems. The
scale is designed to be an indicator for system designs that may or may not include explicit
explanations. At the same time, the SIPA scale is especially suited to e.g., compare different
interfaces or evaluate the development of users’ experience over time. Results of the SIPA scale
should not be interpreted without reference (i.e. a control condition or certain points in time).
Recommended Co-Measures:
Besides SIPA the following measures can be useful for the further interpretation of a user’s
experience as they target constructs relevant to a systems design which are adjacent to SIPA :
• ATI - Affinity for Technology Interaction (Franke et al., 2019)
• FOST - Facets Of System Trustworthiness (Trommler et al., 2018)
• TiA - Trust in Automation (Jian et al., 2000)
References:
Endsley, M. R. (1988). Situation awareness global assessment technique (SAGAT). Proceedings of the
IEEE 1988 National Aerospace and Electronics Conference, 789–795.
https://doi.org/10.1109/NAECON.1988.195097
Franke, T., Attig, C., & Wessel, D. (2019). A Personal Resource for Technology Interaction:
Development and Validation of the Affinity for Technology Interaction (ATI) Scale.
International Journal of Human–Computer Interaction, 35(6), 456–467.
https://doi.org/10.1080/10447318.2018.1456150
Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000). Foundations for an Empirically Determined Scale of
Trust in Automated Systems. International Journal of Cognitive Ergonomics, 4(1), 53–71.
https://doi.org/10.1207/S15327566IJCE0401_04
Schrills, T., Zoubir, M., Bickel, M., Kargl, S., & Franke, T. (2021). Are Users in the Loop? Development
of the Subjective Information Processing Awareness Scale to Assess XAI. ACM CHI Workshop
on Operationalizing Human-Centered Perspectives in Explainable AI. CHI’21, Yokohama,
Japan.
Trommler, D., Attig, C., & Franke, T. (2018). Trust in activity tracker measurement and its link to user
acceptance. https://doi.org/10.18420/MUC2018-MCI-0361
Are Users in the Loop? Development of the Subjective Information Processing Awareness Scale to Assess XAI
Conference Paper
Full-text available
  • May 2021
While interacting with intelligent systems, users need to rely on automated information processing. However, systems often do not offer sufficient transparency to keep users in the loop. Based on Situation Awareness Theory we constructed the 12-item SIPA scale. SIPA assesses if during system interaction users experience themselves as enabled to 1) perceive the information used by the system, 2) understand the system's information processing and 3) predict its outcome. In an online study with N = 55, two systems a web search and an online-symptom checker-were evaluated. Results indicate that the scale is highly reliable, shows moderate to strong correlations with the Trust in Automation scale [21] and the Affinity for Technology Interaction (ATI) Scale [10]. We also found a moderate correlation with the Explanation Satisfaction Scale [17]. Based on the results, the questionnaire appears to be a promising tool to guide future human-centered XAI research.
View
Trust in activity tracker measurement and its link to user acceptance
Conference Paper
Full-text available
  • Sep 2018
Inaccuracies in data measurement can impair trustworthiness of activity trackers (i.e., wearable fitness devices) and, thus, constitute a usability challenge possibly impairing user acceptance. With the present research, we aim at advancing understanding of perceived trustworthiness of activity tracker measurement and its relevance for user acceptance. N = 79 users of activity trackers were surveyed regarding their daily interaction with the tracker, user experience, and user acceptance. Results indicated a substantial variance in perceived trustworthiness. Many users perceived suboptimal trustworthiness of the tracking of their activity data, indicating potential for optimization. Further, analyses showed that higher perceived trustworthiness of activity tracker measurement was indeed linked to higher user acceptance. The results highlight the potential of enhancing user acceptance of activity trackers by improving perceived trustworthiness of activity measurement.
View
Foundations for an Empirically Determined Scale of Trust in Automated Systems
Article
Full-text available
  • Mar 2000
  • Int J Cognit Ergon
One component in the successful use of automated systems is the extent to which people trust the automation to perform effectively. In order to understand the relationship between trust in computerized systems and the use of those systems, we need to be able to effectively measure trust. Although questionnaires regarding trust have been used in prior studies, these questionnaires were theoretically rather than empirically generated and did not distinguish between three potentially different types of trust: human-human trust, human-machine trust, and trust in general. A 3-phased experiment, comprising a word elicitation study, a questionnaire study, and a paired comparison study, was performed to better understand similarities and differences in the concepts of trust and distrust, and among the different types of trust. Results indicated that trust and distrust can be considered opposites, rather than different concepts. Components of trust, in terms of words related to trust, were similar across the three types of trust. Results obtained from a cluster analysis were used to identify 12 potential factors of trust between people and automated systems. These 12 factors were then used to develop a proposed scale to measure trust in automation.
View
A Personal Resource for Technology Interaction: Development and Validation of the Affinity for Technology Interaction (ATI) Scale
Article
  • Mar 2018
Successful coping with technology is relevant for mastering daily life. Based on related conceptions, we propose affinity for technology interaction (ATI), defined as the tendency to actively engage in intensive technology interaction, as a key personal resource for coping with technology. We present the 9-item ATI scale, an economical unidimensional scale that assesses ATI as an interaction style rooted in the construct need for cognition (NFC). Results of multiple studies (n > 1500) showed that the scale achieves good to excellent reliability, exhibits expected moderate to high correlations with geekism, technology enthusiasm, NFC, self-reported success in technical problem-solving and technical system learning success, and also with usage of technical systems. Further, correlations of ATI with the Big Five personality dimensions were weak at most. Based on the results, the ATI scale appears to be a promising tool for research applications such as the characterization of user diversity in system usability tests and the construction of general models of user-technology interaction.
View
Situation global assessment technique (SAGAT)
Conference Paper
  • Jun 1988
Pilot-vehicle interface designs must be driven by the gaol of establishing and maintaining high pilot situation awareness. The situation-awareness global assessment technique (SAGAT), developed to assist in this process by providing an objective measure of pilot's situation awareness with any given aircraft design, is described. SAGAT is considered to represent a substantial improvement in the evaluation of pilot-vehicle interface designs, facilitating the development of cockpits which assist the pilot in surviving combat. A formal definition of situation awareness is presented a description of the SAGAT methology and a discussion of its validation
View