Conference PaperPDF Available

The Impact of Conversational Assistance on the Effective Use of Forecasting Support Systems: A Framed Field Experiment

Authors:

Abstract

Forecasting support systems (FSSs) support demand planners in important forecasting decisions by offering statistical forecasts. However, planners often rely on their judgment more than on system-based advice which can be detrimental to forecast accuracy. This is caused by a lack of understanding and subsequent lack of trust in the FSS and its advice. To address this problem, we explore the potential of extending the traditional static assistance (e.g., manuals, tooltips) with conversational assistance provided by a conversational assistant that answers planners' questions. Drawing on the theory of effective use, we aim to conduct a framed field experiment to investigate whether conversational (vs. static) assistance better supports planners in learning the FSS, increases their trust, and ultimately helps them make more accurate forecasting decisions. With our findings, we aim to contribute to research on FSS design and the body of knowledge on the theory of effective use.
This is the author’s version of a work that was published in the following source:
Haug, S., Ruoff, M., and Gnewuch, U. (2022). The Impact of Conversational Assistance
on the Effective Use of Forecasting Support Systems: A Framed Field Experiment. To
appear in: AIS International Conference on Information Systems (ICIS) 2022, December
11-14, 2022, Copenhagen, Denmark.
Please note: Copyright is owned by the author and / or the publisher. Commercial
use is not allowed.
© 2017. This manuscript version is made available under
the CC-BY-NC-ND 4.0 license
http://creativecommons.org/licenses/by-nc-nd/4.0/
Effective Use of Forecasting Support Systems
Forty-Third International Conference on Information Systems, Copenhagen 2022
1
The Impact of Conversational Assistance on
the Effective Use of Forecasting Support
Systems: A Framed Field Experiment
Short Paper
Saskia Haug
Karlsruhe Institute of Technology
Karlsruhe, Germany
saskia.haug@kit.edu
Marcel Ruoff
Karlsruhe Institute of Technology
Karlsruhe, Germany
marcel.ruoff@kit.edu
Abstract
Forecasting support systems (FSSs) support demand planners in important forecasting
decisions by offering statistical forecasts. However, planners often rely on their judgment
more than on system-based advice which can be detrimental to forecast accuracy. This is
caused by a lack of understanding and subsequent lack of trust in the FSS and its advice.
To address this problem, we explore the potential of extending the traditional static
assistance (e.g., manuals, tooltips) with conversational assistance provided by a
conversational assistant that answers planners’ questions. Drawing on the theory of
effective use, we aim to conduct a framed field experiment to investigate whether
conversational (vs. static) assistance better supports planners in learning the FSS,
increases their trust, and ultimately helps them make more accurate forecasting
decisions. With our findings, we aim to contribute to research on FSS design and the body
of knowledge on the theory of effective use.
Keywords: Effective Use, Conversational Agent, Forecasting Support System, Framed
Field Experiment
Introduction
Demand forecasting is the process of estimating future customer demand over a certain period. It is crucial
for the success of most supply chain-based companies because important internal operational, tactical, and
strategic decisions depend on the results of demand forecasts (Fildes et al. 2006). Accurate forecasting
results have a significant impact on financial savings, the company’s competitiveness, supply chain
relationships, and customer satisfaction (Moon et al. 2003). Information systems that prepare forecasts
and support the forecasting process are referred to as forecasting support systems (FSSs) (Fildes et al.
2006). As a subgroup of decision support systems (DSSs), these systems support demand planners in their
forecasting decisions by offering statistical forecasts, written explanations, or recommendations
(Montazemi et al. 1996; Önkal et al. 2009). In general, FSSs are developed to provide accurate forecasts
that only require human intervention for exceptional circumstances. For example, for regular time series,
the statistical forecasts of FSSs are more accurate than forecasting decisions that are purely based on
planners’ judgment (Fildes et al. 2006).
Effective Use of Forecasting Support Systems
Forty-Third International Conference on Information Systems, Copenhagen 2022
2
However, research has shown that system-generated forecasts are too often replaced by forecasts purely
based on planners’ judgments, which in most cases decreases the effectiveness of the forecasting decision
(Fildes et al. 2009; Goodwin et al. 2013). This might not be surprising since decision-makers in general
tend to trust their own beliefs more than advice from others as they understand their internal reasons better
than those of others (Yaniv & Kleinberger, 2000). Furthermore, humans tend to trust computer-based
advice even less than the advice of other humans (Önkal et al. 2009). The lack of trust in DSS, in general, is
often caused by a lack of understanding and learning (Gönül et al. 2006). Parikh et al. (2001) showed that
assisting users with explanations when using DSS has a positive effect on users’ learning and their final
decision. Existing FSSs often only provide static support in form of manuals or tooltips that do not facilitate
the learning process. Consequently, demand planners need better support in learning the FSS to increase
the effective use of FSSs and ultimately the accuracy of their forecasting decisions. According to learning
theory, humans learn better when having meaningful interactions with others (Kim 2006). While access to
other humans (e.g. colleagues) often is not readily available, technical advances in the field of artificial
intelligence (AI) have allowed the creation of human-like conversational assistants (Diederich et al. 2022).
These conversational assistants can replicate human-like interaction and therefore could fulfill this crucial
need for meaningful interaction through follow-up questions and explanations, similar to how demand
planners would interact with their colleagues. Compared to static support in the form of manuals or tooltips,
AI-based conversational assistants offer more flexibility and can provide more intelligent assistance to
support planners’ learning and effective use of FSSs, which may ultimately increase their trust and help
them make more accurate forecasting decisions.
Therefore, in a joint research project with a multinational chemical company, we aim to investigate how to
support planners learning of an FSS. In the chemical industry, many products are commodities, resulting
in a particularly high pressure on margins and supply chain efficiency (Blackburn et al. 2015). Demand
forecasting in the chemical industry is mainly based on historic demand data and it is very difficult to
consider potential future challenges for the supply chain such as demand volatility, economic uncertainty,
and climate-induced risks (Acar and Gardner 2012). This makes it even more important to develop reliable
FSSs for demand planning in this industry. Our industry partner already developed and implemented an
FSS that provides statistical forecasts for demand planners. However, even though it has been found to
achieve a higher forecast accuracy than human demand planners, the trust in and understanding of the
system remained rather low.
Drawing on the theory of effective use (Burton-Jones and Grange 2013), we primarily seek to investigate
how conversational assistance in FSS impacts planners’ learning. Further, we aim to examine how their
learning actions with the conversational assistant influence their level of effective use and the accuracy of
their forecasting decisions. Moreover, since Burton-Jones and Grange (2013) suggest that there is a
connection between trust and effective use and Goodwin et al. (2013) showed that planners who better
understand their FSS have more trust in the system and make better forecasting decisions, we also aim to
investigate how trust is interwined with the constructs of the theory of effective use. This leads us to the
following research question:
RQ: How does conversational assistance support planners' learning of an FSS, increase their trust in and
effective use of the system, and ultimately help them make more accurate forecasting decisions?
In this paper, we present our research design that draws upon the theory of effective use to analyze how
conversational assistance in FSSs (e.g., explanations and follow-up questions provided by a conversational
assistant) impacts planners’ learning and effective use of the FSS. With our findings, we expect to contribute
to IS literature by offering novel insights on how conversational assistance can improve planners’ learning
of an FSS. In addition, we aim to contribute design knowledge for supporting FSS users with conversational
assistants. Finally, we aim to show that better learning of FSSs affects planners’ trust in the system and that
learning and trust, in turn, positively impact the resulting forecasting decisions.
This paper starts by introducing the theoretical foundations of FSSs for demand planning, conversational
assistants, and the theory of effective use. This is followed by the presentation of the research model,
including the hypotheses. Then, the methodology is described and the artifact that is used for the study is
explained. Finally, we share our expected contributions and give an outlook on this research project.
Effective Use of Forecasting Support Systems
Forty-Third International Conference on Information Systems, Copenhagen 2022
3
Theoretical Foundations
In the following, we introduce the two research fields of FSSs and conversational assistants. Furthermore,
we also provide a short introduction to the theory of effective use.
Forecasting Support Systems for Demand Planning
Accurate demand planning is crucial for operational, tactical, and strategic decisions in companies. To
support human demand planners, forecasting support systems (FSSs) a special class of decision support
systems (DSSs) prepare statistical forecasts to support decision-making processes (Fildes et al. 2006).
Armstrong (2001) defined an FSS formally as “a set of procedures (typically computer-based) that supports
forecasting. It allows the analyst to easily access, organize and analyze a variety of information. It might
also enable the analyst to incorporate judgment and monitor forecast accuracy (p. 8). Therefore, an FSS
usually consists of a database with the time series history, a set of quantitative forecasting techniques, and
facilities that allow planners’ judgment. Planners’ judgment should only be necessary to consider special
factors, such as promotional campaigns and other qualitative information that is not included in the hard
data (Fildes et al. 2006). The two main objectives of using FSSs in the forecasting process are “to improve
the user’s ability to realize when judgmental intervention is appropriate and to enable the user to apply
accurate judgmental interventions when these are appropriate” (Fildes et al. 2006, p. 354). Lawrence et al.
(1986) found that on average a combination of both, statistical forecast and planners’ judgment is better
than either of them alone.
Particularly in the chemical industry, demand forecasts usually rely on historical data (Acar and Gardner
2012). Although the FSS leads on average to a higher forecast accuracy based on such data than planners
judgments without using the FSS, planners often do not rely on the results. The antecedents and effects of
judgmental adjustments to statistical forecasts were recently studied in various contexts (De Baets and
Harvey 2020; Eroglu and Sanders 2022; Lin 2019). A common problem of FSSs is planners’ lack of trust in
the system-generated forecasts and the FSS itself (Goodwin et al. 2013). A lack of trust leads to planners
changing the statistical forecasts provided by the FSS more often, which in turn has a detrimental effect on
forecast accuracy (Goodwin et al. 2013). Existing studies show that a lack of trust can be mitigated by
offering different forms of explanations that positively affect planners’ understanding and learning of the
FSS (Goodwin et al. 2013). Explanations in FSSs can address three goals: explain a perceived anomaly,
supply additional knowledge, and facilitate learning from the system (Gönül et al. 2006). Consequently,
planners require support in the form of contextualized assistance to understand and use FSSs effectively in
their decision-making processes.
Conversational Assistants
Users are able to converse with conversational assistants (CAs) (also referred to as conversational agents,
chatbots, or virtual assistants) either through written or spoken natural language in a turn-by-turn fashion
(Diederich et al. 2022). While the underlying technology of natural language processing has improved
greatly over the last decades, conversational assistants are not a new concept. Joseph Weizenbaum (1966)
already introduced the idea of enabling users to interact through natural language with technological
artifacts in the 1960s by developing ELIZA. However, only recently have conversational assistants been
introduced and researched in various application areas, such as customer service (Gnewuch et al. 2017),
financial advisory (Morana et al. 2020), health care (Prakash and Das 2020), and data analytics (Ruoff and
Gnewuch 2021a).
Organizations are often introducing conversational assistants in their information systems and processes
for two reasons. First, due to their ability of natural language interaction they provide users intuitive access
to existing information systems (McTear et al. 2016) Second, interacting with conversational assistants can
provide the feeling of interacting with a human (Seeger et al. 2018; Verhagen et al. 2014). Especially in the
context of decision support, conversational assistants are promising as they are able to “aid, assist and
advise people in personal and organizational decision situations” (Power et al. 2019, p. 1). However, while
recent studies aim to investigate how conversational assistants influence the effective use of (Ruoff and
Gnewuch 2021b) and trust in (Seeger et al. 2017) an information system, it is not well understood how
conversational assistance influences these constructs and their interplay.
Effective Use of Forecasting Support Systems
Forty-Third International Conference on Information Systems, Copenhagen 2022
4
Theory of Effective Use
Almost a decade ago, Burton-Jones and Grange (2013) proposed the theory of effective use. This theory
explains the nature and drivers of effective use, which is defined as “using a system in a way that helps attain
the goals for using the system” (Burton-Jones and Grange 2013, p. 4). Effective use is conceptualized as an
aggregate construct comprising three hierarchical dimensions: transparent interaction, representational
fidelity, and informed action (see Table 1). Thus, the user’s overall level of effective use is determined by
her/his aggregated levels of the three dimensions. However, each lower-level dimension is necessary but
not sufficient for the higher-level dimension. For example, if users cannot transparently access the system’s
representations, they are unlikely to obtain faithful representations during use and therefore take informed
actions. In contrast, when users achieve a high level of effective use, they are more effective and efficient in
their overall task performance (Burton-Jones and Grange 2013).
In addition, Burton-Jones and Grange (2013) identify two major drivers of effective use: learning and
adaptation. Users can take various learning and adaptation actions to improve their level of effective use.
On the one hand, users can adapt a system to improve its representations or the users’ access to them (e.g.,
by personalizing the interface). On the other hand, users can take learning actions (see Table 1) to improve
their understanding of the system itself or how to obtain and leverage its representations in order to use a
system more effectively. For example, users can attend training sessions or study the system’s manual.
However, the underlying assumption is that learning actions are primarily driven by the user and the system
takes a passive role. More specifically, the system offers ways to learn the system, but it does not actively
encourage users to do so. In light of the recent technological advances, we believe that today’s systems can
take a more active role in learning actions. Therefore, we focus on how conversational assistants can support
users in learning the system, fidelity, and leveraging its representations.
Construct
Definition
Measurements
Learning
Actions
Learning the
System
“any action a user takes to learn the system (its
representations, or its surface or physical structure)”
(Burton-Jones and Grange 2013)
Survey data (Trieu et
al. 2021), chat and log
data
Learning Fidelity
“any action a user takes to learn the extent to which
the output from the system faithfully represents the
relevant real-world domain” (Burton-Jones and
Grange 2013)
Survey data (Trieu et
al. 2021), chat and log
data
Learning to
Leverage
Representations
“any action a user takes to learn how to leverage the
output obtained from the system in his/her work”
(Burton-Jones and Grange 2013)
Survey data (Trieu et
al. 2021), chat and log
data
Effective
Use
Transparent
Interaction
“the extent to which a user is accessing the system's
representations unimpeded by its surface and physical
structures” (Burton-Jones and Grange 2013)
Survey data (Trieu et
al. 2021)
Representational
Fidelity
“the extent to which a user is obtaining
representations from the system that faithfully reflect
the domain being represented” (Burton-Jones and
Grange 2013)
Survey data (Trieu et
al. 2021)
Informed Action
“the extent to which a user acts upon the faithful
representations he or she obtains from the system to
improve his or her state” (Burton-Jones and Grange
2013)
Survey data (Trieu et
al. 2021)
Trust
Trust refers to a willingness to be vulnerable to
another entity (Rousseau et al. 1998)
Survey data (Cyr et al.
2009; Turel et al.
2008)
Effectiveness
“A dimension of performance referring to the extent
to which a user has attained the goals of the task for
which the system was used.” (Burton-Jones and
Grange 2013)
Difference between
planners judgment
and the gold standard
Table 1. Definitions of Key Constructs and Measurements in this Study
Effective Use of Forecasting Support Systems
Forty-Third International Conference on Information Systems, Copenhagen 2022
5
Finally, Burton-Jones and Grange (2013) only briefly touch upon the concept of trust, which is particularly
important in the context of FSSs where users need to rely on the system forecasts to make important job
decisions. Burton-Jones and Grange (2013) state that “representational fidelity should engender feelings of
trust” (p. 653). More specifically, “when representational fidelity increases, users are more likely to trust
their systems because they will have more positive expectations of the consequences of relying on those
representations” (p. 653). Despite these considerations, the theory of effective use and trust so far have not
been integrated and were only studied in isolation. Given that trust plays a crucial role in the context of
FSSs, we also address this gap by examining how trust mediates the relationship between representations
fidelity and effectiveness.
Research Model
The primary focus of this research is to investigate the relationship between the type of assistance an FSS
provides (i.e., conversational vs. static) and the extent to which demand planners engage in learning.
Furthermore, we aim to understand how their level of representational fidelity influences their trust and
how their trust in turn affects the effectiveness of using the FSS. Our research model which is based on the
theory of effective use by Burton-Jones & Grange (2013) is shown in Figure 1. Although the original theory
of effective use includes more relationships, we focus only on those relationships that directly relate to our
hypothesized effects.
Figure 1. Research Model
The theory of effective use distinguishes three different types of learning actions that users can take to
improve their level of effective use: (1) learning the system, (2) learning fidelity, and (3) learning to leverage
representations (Burton-Jones and Grange 2013; Trieu et al. 2021). Traditional FSSs enable these learning
actions by providing static assistance to users in the form of user manuals and tooltips. Planners can then
search for the required information and explanations themselves during the interaction with the FSS.
However, according to learning theory, humans can learn better when having meaningful interactions with
others (Kim 2006). Conversational assistants can mimic human-like interaction and therefore could fulfill
this crucial need for meaningful interaction through follow-up questions and explanations (Diederich et al.
2022). As conversational assistants are more flexible in supporting planners than simple textual support,
we argue that the presence of a conversational assistant positively impacts the extent of user learning.
Accordingly, we propose our first hypotheses (H1, H2, and H3) as follows:
H1, H2, H3: Planners who interact with an FSS that provides conversational (vs. static) assistance engage
more in learning the system (H1), learning fidelity (H2), and learning to leverage representations (H3).
Representational fidelity describes the extent to which planners obtain representations from the system as
faithful (Burton-Jones and Grange 2013). As faith and trust are directly connected (Mcknight et al. 2011),
we propose that representational fidelity has a positive impact on planners’ trust in the system. This
relationship was also already assumed by Burton-Jones and Grange (2013) who state that “representational
fidelity should engender feelings of trust” (p. 653). More specifically, they argue that increasing
representational fidelity will lead to users having more positive expectations when relying on the system’s
Effective Use of Forecasting Support Systems
Forty-Third International Conference on Information Systems, Copenhagen 2022
6
representations which in turn affects their trust in the system. Therefore, we formulate our fourth
hypothesis:
H4: A higher level of representational fidelity when using an FSS increases planners’ trust in the FSS.
Burton-Jones and Grange (2013) also suggest that trust in the system has a positive impact on the
effectiveness of the system usage. When planners trust the system, they are more willing to rely on the
forecasts generated by it. Given that these system-generated forecasts are typically more accurate than the
planners' judgment, it is likely that planners can then make better forecasting decisions. In our context, we
therefore propose that when users trust the system and its representations, they are more likely to consider
the information provided in the FSS in their final decision which should lead to a more accurate forecasting
decision. This leads to H5:
H5: A higher level of planners trust in the FSS leads to higher effectiveness when using the FSS.
Methodology
Experimental Design. We plan to conduct a framed field experiment to empirically test our research model.
To enhance the external validity of the experiment, we developed a custom FSS similar to the one developed
and used by our industry partner, a multinational chemical company. Our custom FSS contains the core
functionalities of the FSS of our industry partner, but some very specific functionalities and settings were
left out to not introduce any confounding factors caused by too complex functionalities and ensure internal
validity. We use a between-subjects design with two conditions (conversational vs. static assistance). The
control group will be provided with the FSS that only provides static assistance in the form of an integrated
manual and tooltips. The treatment group will use the same FSS but with additional conversational
assistance provided by a conversational assistant. A screenshot of the FSS with conversational assistance is
shown in Figure 2 and its user interface (UI) and functionalities are described below.
Artifact. In both conditions, the FSS contains two main views: an overview table that shows all products
(e.g., paint) that are managed by the current user and some additional details like the country, the
segmentation, and the seasonality. The second screen provides a deep dive into the forecast of a specific
product. On top, a graph is shown that displays the historical actual demand, the forecasted demands, and
the future open orders (1). Below, three tabs are included. The first one shows the numbers that belong to
the main graph. The second one offers additional information on the product and the forecasts (2). In the
third tab, the tested methods are listed and information on forecast accuracy and forecast errors is
presented. For assistance on the UI, the FSS includes a manual that can be accessed via the information
icon in the upper right corner (3). This manual provides information on how to interact with the UI (e.g.,
how to zoom in on a specific period in the graph) and where to find which information. To understand the
representations in the UI (e.g., open orders), explanations for specific terms and information can be
accessed via tooltips (4). In addition, the treatment group includes a text-based conversational assistant
that can be accessed via the chat icon in the lower right corner (5).
The conversational assistant supports the three learning actions proposed in the theory of effective use as
follows. For enabling users to learn the system, the conversational assistant can explain to planners where
to find a piece of specific information and can explain what the information and terms that are presented
in the FSS mean. To address the learning of representations, the conversational assistant can explain to
planners how they can check if the information and data presented are accurate. Finally, to support
planners in learning to leverage representations, the conversational assistant can answer questions on
processing and using the information of the FSS to fulfill the planning task. When starting the FSS, the
conversational assistant proactively offers questions planners could ask. Furthermore, it suggests follow-
up questions regarding related terms or questions that address the same term but another type of learning.
Experimental Task. Each participant is randomly assigned to one of the two experimental conditions. After
receiving a short introduction to the task and the FSS, participants read additional information on their
product like the market trend, historical data, and the prior accuracy of the forecast and the planning
decision. Subsequently, they are tasked with forecasting the demand for their product for the next six
months.
Participants. We will recruit the participants for our study at our industry partner. These participants are
actual employees who work in the supply chain management departments and are thereby familiar with the
Effective Use of Forecasting Support Systems
Forty-Third International Conference on Information Systems, Copenhagen 2022
7
task, the FSS, and the related information and terms. Participation will be voluntary, and participants will
be incentivized by receiving an overview of their forecasting performance compared to other participants.
To avoid bias effects, individuals who were involved in the experimental design or participated in initial
exploratory interviews are excluded from the study.
Measurement. For the evaluation, we use both survey and log data. The measurements for each construct
are listed in Table 1. We adopt established and validated measurement items provided by Trieu et al. (2021)
for measuring learning actions, transparent interaction, representational fidelity, and informed action.
Additionally, we analyze the log data of UI interactions and chat messages. By analyzing the questions that
participants ask the CA and the tooltips they access during the experiment, we can measure how extensively
they engaged in each learning action. For measuring trust, we use five items from existing studies (e.g., Cyr
et al. (2009), Turel et al. (2008)). By doing so, we can ensure that the trust items fit our context and our
system. Effectiveness is defined by the accuracy of the forecasting decision and measured by comparing six
planning values that participants must enter to a gold standard based on historical data, which is usually
the actual demand for the respective month. In some exceptional cases, e.g., when the statistical forecast
considered open orders that increased the statistical forecast to a value that is higher than the actual
forecast, the gold standard is represented by the values of the open orders. For being able to compare the
planned values with the actual demand, we present to participants data from the previous year. We are
controlling for multiple demographic characteristics (age, gender, location) and further characteristics of
participants that might impact the results (trusting stance, experience with CAs, experience with the FSS,
experience with demand forecasting in general) as well as characteristics of the FSS like the forecasting
accuracy for the specific forecast.
Analysis. We will conduct a manipulation check to verify the effectiveness of our treatment assignment by
collecting data on whether or not participants in the treatment group interacted with our conversational
assistant. Subsequently, we will test our research model using structural equation modeling.
Figure 2. FSS Artifact used in the Experiment (Treatment Group shown) with (1) Graph
with Forecast Data, (2) Additional Information on Forecast, (3) Manual and (4) Tooltips
for Static Assistance, and (5) Conversational Assistance.
Discussion and Outlook
The effective use of FSSs is of critical importance for many supply chain-based organizations because
forecasting decisions have a major impact on business outcomes such as financial savings, the company’s
competitiveness, supply chain relationships, and customer satisfaction (Moon et al. 2003). However,
demand planners often struggle to understand forecasts provided by an FSS, do not trust them, and end up
Effective Use of Forecasting Support Systems
Forty-Third International Conference on Information Systems, Copenhagen 2022
8
adjusting them mistakenly (Fildes et al. 2009; Goodwin et al. 2013). In our study, we address this challenge
by investigating whether and how extending traditional FSSs with a conversational assistant that supports
planners in learning the system and its representations and leveraging these representations can help
planners to trust the system, use the FSS more effectively, and make more accurate decisions.
Our next step is to conduct our framed field experiment and collect data at our industry partner. We have
already developed the custom FSS and the conversational assistant, and the experiment is planned to be
conducted in the next months. With our findings, we plan to contribute to research on FSS design and the
body of knowledge on the theory of effective use. From a theoretical perspective, we aim to show how
combining a traditional FSS with a conversational assistant that supports planners in learning the FSS can
increase effective use of FSSs. In addition, we aim to contribute design knowledge for supporting FSS users
with conversational assistance. Furthermore, we aim to contribute by offering novel insights on how
learning of FSSs affects planners’ trust in the system and how learning and trust, in turn, impact the
accuracy of the resulting forecasting decisions. The results will help to design better and more effective
FSSs. From a practical perspective, the results of our framed field experiment will help our industry partner
and other practitioners to understand how to support demand planners in learning the FSS and more
specifically how to provide conversational assistance in an FSS. The conversational assistant that we
developed will serve as a starting point for our industry partner to develop a fully operational conversational
FSS.
Although a framed field experiment has many strengths such as collecting data in a real-world context with
actual demand planners, this type of experiment also has its limitations. Our research model will only be
tested in a particular scenario and our findings could be influenced by specific characteristics of our industry
partner or their FSS which inspired the artifact for our experiment. Furthermore, our study only considers
the learning actions described in the theory of effective use, not the adaptation actions. Similarly, we only
investigate the effectiveness but not the efficiency of using the FSS as a dependent variable. Therefore,
future research could expand upon our study by investigating how planners could be supported in adapting
the FSS and how these adaptation actions affect their forecasting decisions. Moreover, it would also be
interesting to explore how conversational assistance impacts the efficiency of demand planning.
References
Acar, Y., and Gardner, E. S. 2012. “Forecasting Method Selection in a Global Supply Chain,” International
Journal of Forecasting (28), pp. 842848.
Armstrong, J. S. 2001. “Principles of Forecasting,” Journal of Marketing Research (Vol. 30), International
Series in Operations Research & Management Science, (J. S. Armstrong, ed.), Boston, MA: Springer
US.
De Baets, S., and Harvey, N. 2020. “Using Judgment to Select and Adjust Forecasts from Statistical
Models,” European Journal of Operational Research (284:3), Elsevier B.V., pp. 882895.
Blackburn, R., Lurz, K., Priese, B., Göb, R., and Darkow, I. L. 2015. “A Predictive Analytics Approach for
Demand Forecasting in the Process Industry,International Transactions in Operational Research
(22:3), pp. 407428.
Burton-Jones, A., and Grange, C. 2013. “From Use to Effective Use: A Representation Theory Perspective,”
Information Systems Research (24:3), INFORMS, pp. 632658.
Cyr, D., Head, M., Larios, H., and Pan, B. 2009. “Exploring Human Images in Website Design: A Multi-
Method Approach,” MIS Quarterly: Management Information Systems (33:3), Management
Information Systems Research Center, pp. 539566.
Diederich, S., Benedikt Brendel, A., Morana, S., Kolbe, L., and Benedikt, A. 2022. “On the Design of and
Interaction with Conversational Agents: An Organizing and Assessing Review of Human-Computer
Interaction Research,” Journal of the Association for Information Systems (23:1), Association for
Information Systems, pp. 96138.
Eroglu, C., and Sanders, N. R. 2022. “Effects of Personality on the Efficacy of Judgmental Adjustments of
Statistical Forecasts,” Management Decision (60:3), Emerald Group Holdings Ltd., pp. 589605.
Fildes, R., Goodwin, P., and Lawrence, M. 2006. “The Design Features of Forecasting Support Systems and
Their Effectiveness,” Decision Support Systems (42:1), North-Holland, pp. 351361.
Fildes, R., Goodwin, P., Lawrence, M., and Nikolopoulos, K. 2009. “Effective Forecasting and Judgmental
Adjustments: An Empirical Evaluation and Strategies for Improvement in Supply-Chain Planning,”
Effective Use of Forecasting Support Systems
Forty-Third International Conference on Information Systems, Copenhagen 2022
9
International Journal of Forecasting (25:1), Elsevier, pp. 323.
Gnewuch, U., Morana, S., and Mädche, A. 2017. “Towards Designing Cooperative and Social Conversational
Agents for Customer Service,” ICIS 2017 Proceedings.
Gönül, M. S., Önkal, D., and Lawrence, M. 2006. “The Effects of Structural Characteristics of Explanations
on Use of a DSS,” Decision Support Systems (42:3), North-Holland, pp. 14811493.
Goodwin, P., Sinan Gönül, M., and Önkal, D. 2013. “Antecedents and Effects of Trust in Forecasting
Advice,” International Journal of Forecasting (29:2), Elsevier B.V., pp. 354366.
Kim, B. 2006. Social Constructivism. Emerging Perspectives on Learning, Teaching, and Technology .
Lawrence, M. J., Edmundson, R. H., and O’Connor, M. J. 1986. “The Accuracy of Combining Judgemental
and Statistical Forecasts,” Management Science (32:12), INFORMS, pp. 15211532.
Lin, V. S. 2019. “Judgmental Adjustments in Tourism Forecasting Practice: How Good Are They?,” Tourism
Economics (25:3), SAGE Publications Inc., pp. 402424.
Mcknight, D. H., Carter, M., Thatcher, J. B., and Clay, P. F. 2011. “Trust in a Specific Technology,” ACM
Transactions on Management Information Systems (TMIS) (2:2), ACM PUB27 New York, NY, USA,
p. 25.
McTear, M., Callejas, Z., and Griol, D. 2016. “The Conversational Interface: Talking to Smart Devices,” The
Conversational Interface: Talking to Smart Devices, Springer International Publishing, pp. 1422.
Montazemi, A. R., Wang, F., Nainar, S. M. K., and Bart, C. K. 1996. “On the Effectiveness of Decisional
Guidance,” Decision Support Systems (18:2), North-Holland, pp. 181198.
Moon, M. A., Mentzer, J. T., and Smith, C. D. 2003. “Conducting a Sales Forecasting Audit,” International
Journal of Forecasting (19:1), Elsevier, pp. 525.
Morana, S., Gnewuch, U., Jung, D., and Granig, C. 2020. “The Effect of Anthropomorphism on Investment
Decision-Making with Robo-Advisor Chatbots,” ECIS 2020 Research Papers.
Önkal, D., Goodwin, P., Thomson, M., Gönül, S., and Pollock, A. 2009. “The Relative Influence of Advice
from Human Experts and Statistical Methods on Forecast Adjustments,” Journal of Behavioral
Decision Making (22:4), John Wiley & Sons, Ltd, pp. 390409.
Parikh, M., Fazlollahi, B., and Verma, S. 2001. “The Effectiveness of Decisional Guidance: An Empirical
Evaluation,” Decision Sciences (32:2), Decision Sciences Institute, pp. 303332.
Power, D., Kaparthi, S., and Mann, A. 2019. “Building Decision Adviser Bots,” MWAIS 2019 Proceedings.
Prakash, A. V., and Das, S. 2020. “Would You Trust a Bot for Healthcare Advice? An Empirical
Investigation,” PACIS 2020 Proceedings.
Rousseau, D. M., Sitkin, S. B., Burt, R. S., and Camerer, C. 1998. “Not so Different After All: A Cross-
Discipline View of Trust,” Academy of Management Review, Academy of Management Briarcliff
Manor, NY 10510, pp. 393404.
Ruoff, M., and Gnewuch, U. 2021a. “Designing Multimodal BI&A Systems for Co-Located Team
Interactions,” ECIS 2021 Research Papers.
Ruoff, M., and Gnewuch, U. 2021b. “Designing Conversational Dashboards for Effective Use in Crisis
Response,” ECIS 2021 Research-in-Progress Papers.
Seeger, A.-M., Pfeiffer, J., and Heinzl, A. 2017. “When Do We Need a Human? Anthropomorphic Design
and Trustworthiness of Conversational Agents,” SIGHCI 2017 Proceedings.
Seeger, A.-M., Pfeiffer, J., and Heinzl, A. 2018. “Designing Anthropomorphic Conversational Agents:
Development and Empirical Evaluation of a Design Framework,” ICIS 2018 Proceedings.
Trieu, V.-H., Burton-Jones, A., Green, P., and Cockcroft, S. 2021. “Applying and Extending the Theory of
Effective Use in a Business Intelligence Context,” MIS Quarterly, Forthcoming.
Turel, O., Yuan, Y., and Connelly, C. E. 2008. “In Justice We Trust: Predicting User Ucceptance of E-
Customer Services,” Journal of Management Information Systems (24:4), Routledge, pp. 123151.
Verhagen, T., van Nes, J., Feldberg, F., and van Dolen, W. 2014. “Virtual Customer Service Agents: Using
Social Presence and Personalization to Shape Online Service Encounters,” Journal of Computer-
Mediated Communication (19:3), John Wiley & Sons, Inc New York, NY, USA, pp. 529545.
Weizenbaum, J. 1966. “ELIZA - A Computer Program for the Study of Natural Language Communication
between Man and Machine,” Communications of the ACM (9:1), ACM PUB27 New York, NY, USA, pp.
3645.
ResearchGate has not been able to resolve any citations for this publication.
Conference Paper
Full-text available
Teams are crucial for organizations in making data-driven decisions. However, current business intelligence & analytics (BI&A) systems are primarily designed to support individuals and, therefore, cannot be used effectively in co-located team interactions. To address this challenge, we conduct a design science research (DSR) project to design a multimodal BI&A system providing touch and speech interactions that can be used effectively by teams. Drawing on the theory of effective use and existing guidelines for multimodal user interfaces, we propose three design principles and instantiate them in a software artifact. The results of a focus group evaluation indicate that enhancing the BI&A system with multimodal capabilities increases transparent interaction and facilitates effective use of the system in co-located team interactions. Our DSR project contributes novel design knowledge for multimodal BI&A systems with touch and speech modalities that facilitate effective use in co-located team interactions
Conference Paper
Full-text available
Dashboards are increasingly used by governments and health organizations to provide important information to the general public during a crisis. However, in contrast to organizational settings, the majority of the general population has not or rarely used dashboards before and therefore often struggles to interact effectively with these dashboards. To address this challenge, we conduct a design science research (DSR) project to design a conversational dashboard that enables natural language-based interactions to facilitate its effective use. Drawing on the theory of effective use, our DSR project aims to provide theory-grounded design knowledge for conversational dashboards that help users to access and find information via natural language. Moreover, we seek to provide novel insights that support researchers and practitioners in understanding and designing more natural and effective interactions between users and dashboards.
Article
Full-text available
Conversational agents (CAs), described as software with which humans interact through natural language, have increasingly attracted interest in both academia and practice, due to improved capabilities driven by advances in artificial intelligence and, specifically, natural language processing. CAs are used in contexts like people's private life, education, and healthcare, as well as in organizations, to innovate and automate tasks, for example in marketing and sales or customer service. In addition to these application contexts, such agents take on different forms concerning their embodiment, the communication mode, and their (often human-like) design. Despite their popularity, many CAs are not able to fulfill expectations and to foster a positive user experience is a challenging endeavor. To better understand how CAs can be designed to fulfill their intended purpose, and how humans interact with them, a multitude of studies focusing on human-computer interaction have been carried out. These have contributed to our understanding of this technology. However, currently a structured overview of this research is missing, which impedes the systematic identification of research gaps and knowledge on which to build on in future studies. To address this issue, we have conducted an organizing and assessing review of 262 studies, applying a socio-technical lens to analyze CA research regarding the user interaction, context, agent design, as well as perception and outcome. We contribute an overview of the status quo of CA research, identify four research streams through a cluster analysis, and propose a research agenda comprising six avenues and sixteen directions to move the field forward.
Conference Paper
Full-text available
Robo-advisors promise to offer professional financial advice to private households at low cost. However, the automation of financial advisory processes to fully-digitalized services often goes hand in hand with a loss of human contact and lack of trust. Especially in an investment context, customers demand a human involved to ensure the trustworthiness of digitalized financial services. In this paper, we report the findings of a lab experiment (N=183) investigating how different levels of anthropomorphic design of a robo-advisor chatbot might compensate for the lack of human involvement and positively affect users’ trusting beliefs and likeliness to follow its recommendations. We found that the anthropomorphic design influences users’ perceived social presence of the chatbot. While trusting beliefs in the chatbot enhance users’ likeliness to follow the recommendation by the chatbot, we do not find evidence for a direct effect of social presence on likeliness to follow its recommendation. However, social presence has a positive indirect effect on likeliness to follow the recommendation via trusting beliefs. Our research contributes to the body of knowledge on the design of robo-advisors and provides more insight into the factors that influence users’ investment decisions when interacting with a robo-advisor chatbot.
Conference Paper
Full-text available
The idea of interacting with computers through natural language dates back to the 1960s, but recent technological advances have led to a renewed interest in conversational agents such as chatbots or digital assistants. In the customer service context, conversational agents promise to create a fast, convenient, and cost-effective channel for communicating with customers. Although numerous agents have been implemented in the past, most of them could not meet the expectations and disappeared. In this paper, we present our design science research project on how to design cooperative and social conversational agents to increase service quality in customer service. We discuss several issues that hinder the success of current conversational agents in customer service. Drawing on the cooperative principle of conversation and social response theory, we propose preliminary meta-requirements and design principles for cooperative and social conversational agents. Next, we will develop a prototype based on these design principles.
Article
Purpose The purpose of this paper is to investigate the effects of personality dimensions (conscientiousness, neuroticism, extraversion, agreeableness, openness to experience, locus of control) on the efficacy of judgmental adjustments of statistical forecasts. Design/methodology/approach This paper uses a two-level hierarchical linear model to analyze a large data set obtained from an organizational setting (a quick service restaurant chain) that includes 3,812 judgmental adjustments of sales forecasts made by 112 store managers. Findings The results indicate that the average forecast accuracy improves as a result of judgmental adjustments, but performance of individual forecasters varies considerably based on their personality. Specifically, the trait of openness to experience tends to improve forecast accuracy while extraversion and external locus of control have negative effects. Originality/value Integration of human judgment with analytics algorithms is a major challenge for organizations. Documenting the impact of these traits on forecast accuracy opens the door for forecasting support system design, training, personnel selection and correction strategies that can be applied to judgmental adjustments.
Article
The benefits that organizations accrue from information systems depend on how effectively the systems are used. Yet despite the importance of knowing what it takes to use information systems effectively, little theory on the topic exists. One recent and largely untested exception is the Theory of Effective Use (TEU). We report on a contextualization, extension, and test of TEU in the business intelligence (BI) context, a context of considerable importance in which researchers have called for such studies. We used a mixed-method, three-phase approach involving instrument development (n = 218), two-wave cross-sectional survey (n = 437), and three sets of follow-up interviews (n = 33). The paper contributes by: 1) showing how TEU can be contextualized, operationalized, and extended, 2) demonstrating that many of TEU’s predictions hold in the BI context while also revealing ways to improve the theory; and 3) offering practical insights executives can draw on to improve use of BI in their organizations.
Conference Paper
Artificial Intelligence (AI)-based automated chatbots are seen as an interactive and scalable way to personalize healthcare-related communication with the information seekers. These chatbots can improve accessibility to healthcare services, by serving as the primary point of contact for the patients. It can also empower patients to take quicker and better healthcare decisions. Additionally, it has the potential to improve efficiency by alleviating the pressure on overly burdened healthcare systems. Despite these benefits, the risks associated with the use of this technology could potentially undermine the user’s trust/confidence in the technology and impede its adoption. In this context, we investigated the factors influencing consumer's trust in AI-based health chatbots using an integrated framework based on Technology Trust Model. The study found empirical evidence for four determinants of consumer’s trust in health chatbots namely, social presence, perceived usefulness, safety risk, and propensity to trust. Implications for theory and practice are discussed. Keywords: Artificial Intelligence, Health Chatbots, Technology Trust, Risk
Article
Forecasting support systems allow users to choose different statistical forecasting methods. But how well do they make this choice? We examine this in two experiments. In the first one (N = 191), people selected the model that they judged to perform the best. Their choice outperformed forecasts made by averaging the model outputs and improved with a larger difference in quality between models and a lower level of noise in the data series. In a second experiment (N = 161), participants were asked to make a forecast and were then offered advice in the form of a model forecast. They could then re-adjust their forecast. Final forecasts were more influenced by models that made better forecasts. As forecasters gained experience, they followed input from high-quality models more readily. Thus, both experiments show that forecasters have ability to use and learn from visual records of past performance to select and adjust model-based forecasts appropriately.
Article
This study aims to evaluate the accuracy of different judgmental forecasting tasks, compare the judgmental forecasting behaviour of tourism researchers and practitioners and explore the validity of experts’ judgmental behaviour by using the Hong Kong visitor arrivals forecasts over the period 2011Q2−2015Q4. Delphi-based judgmental forecasting procedure was employed through the Hong Kong Tourism Demand Forecasting System, an online forecasting support system, to collect and combine experts’ adjusted forecasts. This study evaluates forecasting performance and explores the characteristics of judgmental adjustment behaviour through the use of a group of error measures and statistical tests. The findings suggest a positive correlation between forecast accuracy and the level of data variability, and that experts’ adjustments are more beneficial in terms of achieving higher accuracy for series with higher variability. Industry practitioners’ forecasts outperformed academic researchers, particularly in making short-term forecasts. However, no significant difference was found between the two panels in making directionally correct forecasts. Experts’ judgmental intervention was found most useful for those series most in need of adjustment. The size of adjustment was found to have a strong and significantly positive association with the direction of forecast adjustment, but no statistically significant evidence was found regarding the relationship between accuracy improvement and adjustment size.