No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Value Alignment and Trust in Human-Robot Interaction: Insights from Simulation and User Study
Abstract
With the advent of AI technologies, humans and robots are increasingly teaming up to perform collaborative tasks. To enable smooth and effective collaboration, the topic of value alignment (operationalized herein as the degree of dynamic goal alignment within a task) between the robot and the human is gaining increasing research attention. Prior literature on value alignment makes an inherent assumption that aligning the values of the robot with that of the human benefits the team. This assumption, however, has not been empirically verified. Moreover, prior literature does not account for human’s trust in the robot when analyzing human-robot value alignment. Thus, a research gap needs to be bridged by answering two questions: How does alignment of values affect trust? Is it always beneficial to align the robot’s values with that of the human? We present a simulation study and a human-subject study to answer these questions. Results from the simulation study show that alignment of values is important for trust when the overall risk level of the task is high. We also present an adaptive strategy for the robot that uses Inverse Reinforcement Learning (IRL) to match the values of the robot with those of the human during interaction. Our simulations suggest that such an adaptive strategy is able to maintain trust across the full spectrum of human values. We also present results from an empirical study that validate these findings from simulation. Results indicate that real-time personalized value alignment is beneficial to trust and perceived performance by the human when the robot does not have a good prior on the human’s values.
... Acknowledging that trust is a dynamic variable, several computational trust models in dyadic human-robot teams have been developed (Chen et al., 2018;Xu and Dudek, 2015;Guo and Yang, 2020;Bhat et al., 2024). Notably, Xu and Dudek (2015) proposed the online probabilistic trust inference model (OPTIMo) utilizing Bayesian networks to estimate human trust based on the autonomous agent's performance and human behavioral signals. ...
Trust is a crucial factor for effective human–robot teaming. Existing literature on trust modeling predominantly focuses on dyadic human-autonomy teams where one human agent interacts with one robot. There is little, if not no, research on trust modeling in teams consisting of multiple human and robotic agents. To fill this important research gap, we present the Trust Inference and Propagation (TIP) model to model and estimate human trust in multi-human multi-robot teams. In a multi-human multi-robot team, we postulate that there exist two types of experiences that a human agent has with a robot: direct and indirect experiences. The TIP model presents a novel mathematical framework that explicitly accounts for both types of experiences. To evaluate the model, we conducted a human-subject experiment with 15 pairs of participants (N=30N=30). Each pair performed a search and detection task with two drones. Results show that our TIP model successfully captured the underlying trust dynamics and significantly outperformed a baseline model. To the best of our knowledge, the TIP model is the first mathematical framework for computational trust modeling in multi-human multi-robot teams.
Data generation and labeling are often expensive in robot learning. Preference-based learning is a concept that enables reliable labeling by querying users with preference questions. Active querying methods are commonly employed in preference-based learning to generate more informative data at the expense of parallelization and computation time. In this paper, we develop a set of novel algorithms, batch active preference-based learning methods, that enable efficient learning of reward functions using as few data samples as possible while still having short query generation times and also retaining parallelizability. We introduce a method based on determinantal point processes (DPP) for active batch generation and several heuristic-based alternatives. Finally, we present our experimental results for a variety of robotics tasks in simulation. Our results suggest that our batch active learning algorithm requires only a few queries that are computed in a short amount of time. We showcase one of our algorithms in a study to learn human users’ preferences.
We present the effect of adapting to human preferences on trust in a human-robot teaming task. The team performs a task in which the robot acts as an action recommender to the human. It is assumed that the behavior of the human and the robot is based on some reward function they try to optimize. We use a new human trust-behavior model that enables the robot to learn and adapt to the human's preferences in real-time during their interaction using Bayesian Inverse Reinforcement Learning. We present three strategies for the robot to interact with a human: a non-learner strategy, in which the robot assumes that the human's reward function is the same as the robot's, a non-adaptive learner strategy that learns the human's reward function for performance estimation, but still optimizes its own reward function, and an adaptive-learner strategy that learns the human's reward function for performance estimation and also optimizes this learned reward function. Results show that adapting to the human's reward function results in the highest trust in the robot.
Robots like human co-workers can make mistakes violating a human’s trust in them. When mistakes happen, humans can see robots as less trustworthy which ultimately decreases their trust in them. Trust repair strategies can be employed to mitigate the negative impacts of these trust violations. Yet, it is not clear whether such strategies can fully repair trust nor how effective they are after repeated trust violations. To address these shortcomings, this study examined the impact of four distinct trust repair strategies: apologies, denials, explanations, and promises on overall trustworthiness and its sub-dimensions: ability, benevolence, and integrity after repeated trust violations. To accomplish this, a between-subjects experiment was conducted where participants worked with a robot co-worker to accomplish a task. The robot violated the participant’s trust and then provided a particular repair strategy. Results indicated that after repeated trust violations, none of the repair strategies ever fully repaired trustworthiness and two of its sub-dimensions: ability and integrity. In addition, after repeated interactions, apologies, explanations, and promises appeared to function similarly to one another, while denials were consistently the least effective at repairing trustworthiness and its sub-dimensions. In sum, this paper contributes to the literature on human–robot trust repair through both of these original findings.
Artificial intelligences (AIs) are widely used in tasks ranging from transportation to healthcare and military, but it is not yet known how people prefer them to act in ethically difficult situations. In five studies (an anthropological field study, n = 30, and four experiments, total n = 2150), we presented people with vignettes where a human or an advanced robot nurse is ordered by a doctor to forcefully medicate an unwilling patient. Participants were more accepting of a human nurse's than a robot nurse's forceful medication of the patient, and more accepting of (human or robot) nurses who respected patient autonomy rather than those that followed the orders to forcefully medicate (Study 2). The findings were robust against the perceived competence of the robot (Study 3), moral luck (whether the patient lived or died afterwards; Study 4), and command chain effects (Study 5; fully automated supervision or not). Thus, people prefer robots capable of disobeying orders in favour of abstract moral principles like valuing personal autonomy. Our studies fit in a new era in research, where moral psychological phenomena no longer reflect only interactions between people, but between people and autonomous AIs.
Objective
We examine how human operators adjust their trust in automation as a result of their moment-to-moment interaction with automation.
Background
Most existing studies measured trust by administering questionnaires at the end of an experiment. Only a limited number of studies viewed trust as a dynamic variable that can strengthen or decay over time.
Method
Seventy-five participants took part in an aided memory recognition task. In the task, participants viewed a series of images and later on performed 40 trials of the recognition task to identify a target image when it was presented with a distractor. In each trial, participants performed the initial recognition by themselves, received a recommendation from an automated decision aid, and performed the final recognition. After each trial, participants reported their trust on a visual analog scale.
Results
Outcome bias and contrast effect significantly influence human operators’ trust adjustments. An automation failure leads to a larger trust decrement if the final outcome is undesirable, and a marginally larger trust decrement if the human operator succeeds the task by him/herself. An automation success engenders a greater trust increment if the human operator fails the task. Additionally, automation failures have a larger effect on trust adjustment than automation successes.
Conclusion
Human operators adjust their trust in automation as a result of their moment-to-moment interaction with automation. Their trust adjustments are significantly influenced by decision-making heuristics/biases.
Application
Understanding the trust adjustment process enables accurate prediction of the operators’ moment-to-moment trust in automation and informs the design of trust-aware adaptive automation.
The ability to collaborate with previously unseen human teammates is crucial for artificial agents to be effective in human-agent teams (HATs). Due to individual differences and complex team dynamics, it is hard to develop a single agent policy to match all potential teammates. In this paper, we study both human-human and humanagent teams in a dyadic cooperative task, Team Space Fortress (TSF). Results show that the team performance is influenced by both players’ individual skill level and their ability to collaborate with different teammates by adopting complementary policies. Based on human-human team results, we propose an adaptive agent that identifies different human policies and assigns a complementary partner policy to optimize team performance. The adaptation method relies on a novel similarity metric to infer human policy and then selects the most complementary policy from a pre-trained library of exemplar policies. We conducted human-agent experiments to evaluate the adaptive agent and examine mutual adaptation in humanagent teams. Results show that both human adaptation and agent adaptation contribute to team performance
Trust in autonomous teammates has been shown to be a key factor in human-autonomy team (HAT) performance, and anthropomorphism is a closely related construct that is underexplored in HAT literature. This study investigates whether perceived anthropomorphism can be measured from team communication behaviors in a simulated remotely piloted aircraft system task environment, in which two humans in unique roles were asked to team with a synthetic (i.e., autonomous) pilot agent. We compared verbal and self-reported measures of anthropomorphism with team error handling performance and trust in the synthetic pilot. Results for this study show that trends in verbal anthropomorphism follow the same patterns expected from self-reported measures of anthropomorphism, with respect to fluctuations in trust resulting from autonomy failures.
We introduce a novel capabilities-based bi-directional multi-task trust model that can be used for trust prediction from either a human or a robotic trustor agent. Tasks are represented in terms of their capability requirements, while trustee agents are characterized by their individual capabilities. Trustee agents' capabilities are not deterministic; they are represented by belief distributions. For each task to be executed, a higher level of trust is assigned to trustee agents who have demonstrated that their capabilities exceed the task's requirements. We report results of an online experiment with 284 participants, revealing that our model outperforms existing models for multi-task trust prediction from a human trustor. We also present simulations of the model for determining trust from a robotic trustor. Our model is useful for control authority allocation applications that involve human-robot teams.
Objective
This paper reviews recent articles related to human trust in automation to guide research and design for increasingly capable automation in complex work environments.
Background
Two recent trends—the development of increasingly capable automation and the flattening of organizational hierarchies—suggest a reframing of trust in automation is needed.
Method
Many publications related to human trust and human–automation interaction were integrated in this narrative literature review.
Results
Much research has focused on calibrating human trust to promote appropriate reliance on automation. This approach neglects relational aspects of increasingly capable automation and system-level outcomes, such as cooperation and resilience. To address these limitations, we adopt a relational framing of trust based on the decision situation, semiotics, interaction sequence, and strategy. This relational framework stresses that the goal is not to maximize trust, or to even calibrate trust, but to support a process of trusting through automation responsivity.
Conclusion
This framing clarifies why future work on trust in automation should consider not just individual characteristics and how automation influences people, but also how people can influence automation and how interdependent interactions affect trusting automation. In these new technological and organizational contexts that shift human operators to co-operators of automation, automation responsivity and the ability to resolve conflicting goals may be more relevant than reliability and reliance for advancing system design.
Application
A conceptual model comprising four concepts—situation, semiotics, strategy, and sequence—can guide future trust research and design for automation responsivity and more resilient human–automation systems.
To facilitate effective human-robot interaction (HRI), trust-aware HRI has been proposed, wherein the robotic agent explicitly considers the human's trust during its planning and decision making. The success of trust-aware HRI depends on the specification of a trust dynamics model and a trust-behavior model. In this study, we proposed one novel trust-behavior model, namely the reverse psychology model, and compared it against the commonly used disuse model. We examined how the two models affect the robot's optimal policy and the human-robot team performance. Results indicate that the robot will deliberately ‘manipulate’ the human's trust under the reverse psychology model. To correct this \textcolor{blue}{‘manipulative’} behavior, we proposed a trust-seeking reward function that facilitates trust establishment without significantly sacrificing the team performance.
Trust in automation, or more recently trust in autonomy, has received extensive research attention in the past three decades. The majority of prior literature adopted a "snapshot" view of trust and typically evaluated trust through questionnaires administered at the end of an experiment. This "snapshot" view, however, does not acknowledge that trust is a dynamic variable that can strengthen or decay over time. To fill the research gap, the present study aims to model trust dynamics when a human interacts with a robotic agent over time. The underlying premise of the study is that by interacting with a robotic agent and observing its performance over time, a rational human agent will update his/her trust in the robotic agent accordingly. Based on this premise, we develop a personalized trust prediction model and learn its parameters using Bayesian inference. Our proposed model adheres to three properties of trust dynamics characterizing human agents' trust development process de facto and thus guarantees high model explicability and generalizability. We tested the proposed method using an existing dataset involving 39 human participants interacting with four drones in a simulated surveillance mission. The proposed method obtained a root mean square error of 0.072, significantly outperforming existing prediction methods. Moreover, we identified three distinct types of trust dynamics, the Bayesian decision maker, the oscillator, and the disbeliever, respectively. This prediction model can be used for the design of individualized and adaptive technologies.
Automated vehicles (AVs) that intelligently interact with drivers must build a trustworthy relationship with them. A calibrated level of trust is fundamental for the AV and the driver to collaborate as a team. Techniques that allow AVs to perceive drivers' trust from drivers' behaviors and react accordingly are, therefore, needed for context-aware systems designed to avoid trust miscalibrations. This letter proposes a framework for the management of drivers' trust in AVs. The framework is based on the identification of trust miscalibrations (when drivers' undertrust or overtrust the AV) and on the activation of different communication styles to encourage or warn the driver when deemed necessary. Our results show that the management framework is effective, increasing (decreasing) trust of undertrusting (overtrusting) drivers, and reducing the average trust miscalibration time periods by approximately 40%. The framework is applicable for the design of SAE Level 3 automated driving systems and has the potential to improve the performance and safety of driver-AV teams.
Human trust in automation has been receiving a great interest due to its implications on human reliance on automation, and consequently on mission success. Several computational models have been proposed to quantify trust and its relationship to other system variables. However, these models are still under-utilised in human-machine interaction settings due to the gap between the intent of modellers to capture a phenomenon and the requirements for deploying the model in a practical context. This paper proposes a framework that facilitates the use of computational models for trust in the design and deployment of trust-aware systems. The framework is intended to help designers explore different ways in which computational trust models can be used within the design and use of trust-aware interaction. Besides, the framework is aimed at bridging the gap between the needs of designers and the requirements that should be considered by researchers. Future research directions are identified.
Explanations given by automation are often used to promote automation adoption. However, it remains unclear whether explanations promote acceptance of automated vehicles (AVs). In this study, we conducted a within-subject experiment in a driving simulator with 32 participants, using four different conditions. The four conditions included: (1) no explanation, (2) explanation given before or (3) after the AV acted and (4) the option for the driver to approve or disapprove the AV's action after hearing the explanation. We examined four AV outcomes: trust, preference for AV, anxiety and mental workload. Results suggest that explanations provided before an AV acted were associated with higher trust in and preference for the AV, but there was no difference in anxiety and workload. These results have important implications for the adoption of AVs.
While the question of misspecified objectives has gotten much attention in recent years, most works in this area primarily focus on the challenges related to the complexity of the objective specification mechanism (for example, the use of reward functions). However, the complexity of the objective specification mechanism is just one of many reasons why the user may have misspecified their objective. A foundational cause for misspecification that is being overlooked by these works is the inherent asymmetry in human expectations about the agent's behavior and the behavior generated by the agent for the specified objective. To address this, we propose a novel formulation for the objective misspecification problem that builds on the human-aware planning literature, which was originally introduced to support explanation and explicable behavioral generation. Additionally, we propose a first-of-its-kind interactive algorithm that is capable of using information generated under incorrect beliefs about the agent to determine the true underlying goal of the user.
Trust plays a crucial role in decision-making during human-robot collaboration, particularly in emergency scenarios where it becomes more susceptible due to dynamic factors. Misalignment of human-robot trust significantly hampers the efficiency of the collaboration. Therefore, it is imperative to establish effective human-robot interaction and decision support mechanisms that mitigate biases in human confidence levels regarding the robot’s capabilities in dynamic environments. Additionally, online correction of human-robot trust based on human behavioral feedback is vital. This paper focuses on a specific type of emergency task scenario, specifically, indoor human-robot collaborative patrolling in the event of sudden power outages. We propose a trust model based on linear Gaussian and sparse Gaussian processes (sparse GP). We also employ Monte Carlo Tree Search (MCTS) method to determine the robot’s optimal fusion decision-making. Through VR-based human-robot collaborative experiments, we ascertain that the robot prioritizes enhancing human-robot trust in emergency scenarios to mitigate the long-term costs of human-robot collaboration.
Note to Practitioners
—The primary motivation of this paper lies in tackling the issue of decreased collaboration efficiency in human-robot cooperation, stemming from irrational human decision-making, a problem exacerbated in emergency scenarios where comprehending all available information proves challenging. Grounded in the realm of human-robot trust, this study evaluates the temporal and substantive aspects of human-robot interactions contingent upon the degree of trust. Moreover, it adapts the fusion decision-making process within the human-robot team in accordance with the decision-making strategies adopted by humans at varying trust levels. The optimized decisions of the human-robot ensemble are conveyed to the human participant via decision support. This approach ensures the maintenance of trust levels, facilitating the acceptance of decision support that may seem intuitively irrational but holds objective superiority. Furthermore, it mitigates redundant human-robot interactions once a sufficient level of trust is established.
With an aging population and a growing shortage of caregivers, the need for in-home robots is increasing. However, it is intractable for robots to have all functionalities pre-programmed prior to deployment. Instead, it is more realistic for robots to engage in supplemental, on-site learning about the user's needs and preferences. Such learning may occur in the presence of or involve the user. We investigate the impacts on end-users of in situ robot learning through a series of human-subjects experiments. We examine how different learning methods influence both in-person and remote participants' perceptions of the robot. While we find that the degree of user involvement in the robot's learning method impacts perceived anthropomorphism (p=.001), we find that it is the participants' perceived success of the robot that impacts the participants' trust in (p<.001) and perceived usability of the robot (p<.001) rather than the robot's learning method. Therefore, when presenting robot learning, the performance of the learning method appears more important than the degree of user involvement in the learning. Furthermore, we find that the physical presence of the robot impacts perceived safety (p<.001), trust (p<.001), and usability (p<.014). Thus, for tabletop manipulation tasks, researchers should consider the impact of physical presence on experiment participants.
Trust in automation has been identified as one central factor in effectiveHumans human-autonomy interaction. Despite active research in the past 30 years, most studies have used a “snapshot” view of trust and evaluated trust using questionnairesQuestionnaire administered at the end of an experiment. This “snapshot” view does not fully acknowledge that trust is a dynamicDynamics variable that can strengthen and decay over time. With few exceptions, we have little understanding of the temporal dynamicsDynamics of trust formation and evolution, nor of how trust changes over time as a result of moment-to-moment interactions with autonomy. In this chapter, we present and synthesize the results of two studies examining trust dynamicsDynamics inHumans human-autonomy interaction. In study 1, we identify and define three properties of trust dynamicsDynamics, namely continuity, negativity bias, and stabilization. The three properties characterize a humanHumans agent’s trust formation and evolution process de facto. In study 2, we propose a computational model of trust dynamicsDynamics that adheres to the three properties and evaluate the computational model against existing trust inference models. Results show that our model provides superior predictionPredictions performance, and moreover, guarantees good model explainability and generalizabilityGeneralizability.
In this paper, we present a framework for trust-aware sequential decision-making in a human-robot team wherein the human agent’s trust in the robotic agent is dependent on the reward obtained by the team. We model the problem as a finite-horizon Markov Decision Process with the trust of the human on the robot as a state variable. We develop a reward-based performance metric to drive the trust update model, allowing the robotic agent to make trust-aware recommendations. We conduct a human-subject experiment with a total of 45 participants and analyze how the human agent’s trust evolves over time. Results show that the proposed trust update model is able to accurately capture the human agent’s trust dynamics. Moreover, we cluster the participants’ trust dynamics into three categories, namely, Bayesian decision makers, oscillators, and disbelievers, and identify personal characteristics that could be used to predict which type of trust dynamics a person will belong to. We find that the disbelievers are less extroverted, less agreeable, and have lower expectations toward the robotic agent, compared to the Bayesian decision makers and oscillators. The oscillators tend to get significantly more frustrated than the Bayesian decision makers.
Performance within Human-Autonomy Teams (HATs) is influenced by the effectiveness of communication between humans and robots. Communication is particularly important when robot teammates engage in behaviors that were not anticipated by the human teammate which could degrade trust. However, the literature on trust repair focuses on the role of apologies which may not be appropriate for an unexpected behavior since this behavior may not be an error. Explanations are one method that can be used by robot teammates to avoid costly trust degradation when human expectations are violated. The current study used an Urban Search and Rescue (USAR) scenario to examine the role of explanation in a context wherein a robot teammate deviated from an expected behavior. The current study examined how trust, trustworthiness, and responsibility attribution were influenced by observing the robot teammate deviate from an expected behavior. Participants (N = 148) used an online platform to view videos of the robot: 1) following a planned search path, and 2) deviating from a planned search path. A debriefing event between the human and the robot followed each search activity. Four explanation conditions were tested in the debriefing phase following the behavioral violation. Results showed that trust and trustworthiness (ability, benevolence, and integrity) declined following the unexpected behavior. Accordingly, responsibility attribution shifted from the human to the robot following the unexpected behavior. Explanation strategies that focused on communicating why the event occurred by highlighting the robot's environmental awareness were most effective at buffering the decline in trust and trustworthiness.
A prerequisite for social coordination is bidirectional communication between teammates, each playing two roles simultaneously: as receptive listeners and expressive speakers. For robots working with humans in complex situations with multiple goals that differ in importance, failure to fulfill the expectation of either role could undermine group performance due to misalignment of values between humans and robots. Specifically, a robot needs to serve as an effective listener to infer human users’ intents from instructions and feedback and as an expressive speaker to explain its decision processes to users. Here, we investigate how to foster effective bidirectional human-robot communications in the context of value alignment—collaborative robots and users form an aligned understanding of the importance of possible task goals. We propose an explainable artificial intelligence (XAI) system in which a group of robots predicts users’ values by taking in situ feedback into consideration while communicating their decision processes to users through explanations. To learn from human feedback, our XAI system integrates a cooperative communication model for inferring human values associated with multiple desirable goals. To be interpretable to humans, the system simulates human mental dynamics and predicts optimal explanations using graphical models. We conducted psychological experiments to examine the core components of the proposed computational framework. Our results show that real-time human-robot mutual understanding in complex cooperative tasks is achievable with a learning model based on bidirectional communication. We believe that this interaction framework can shed light on bidirectional value alignment in communicative XAI systems and, more broadly, in future human-machine teaming systems.
Inverse reinforcement learning (IRL) is the problem of inferring the reward function of an agent, given its policy or observed behavior. Analogous to RL, IRL is perceived both as a problem and as a class of methods. By categorically surveying the extant literature in IRL, this article serves as a comprehensive reference for researchers and practitioners of machine learning as well as those new to it to understand the challenges of IRL and select the approaches best suited for the problem on hand. The survey formally introduces the IRL problem along with its central challenges such as the difficulty in performing accurate inference and its generalizability, its sensitivity to prior knowledge, and the disproportionate growth in solution complexity with problem size. The article surveys a vast collection of foundational methods grouped together by the commonality of their objectives, and elaborates how these methods mitigate the challenges. We further discuss extensions to the traditional IRL methods for handling imperfect perception, an incomplete model, learning multiple reward functions and nonlinear reward functions. The article concludes the survey with a discussion of some broad advances in the research area and currently open research questions.
Haptic shared control is used to manage the control authority allocation between a human and an autonomous agent in semi-autonomous driving. Existing haptic shared control schemes, however, do not take full consideration of the human agent. To fill this research gap, this study presents a haptic shared control scheme that adapts to a human operator's workload, eyes on road and input torque in real time. We conducted human-in-the-loop experiments with 24 participants. In the experiment, a human operator and an autonomy module for navigation shared the control of a simulated notional High Mobility Multipurpose Wheeled Vehicle (HMMWV) at a fixed speed. At the same time, the human operator performed a target detection task. The autonomy could be either adaptive or non-adaptive to the above-mentioned human factors. Results indicate that the adaptive haptic control scheme resulted in significantly lower workload, higher trust in autonomy, better driving task performance and smaller control effort.
Proximate interaction is central to a variety of human-robot interaction (HRI) domains including healthcare, manufacturing, and education. While teaming in these domains can improve many aspects of everyday life, the dynamic environments in which it occurs often involve highly variable human behaviors and frequent robot errors. Thus, a well-calibrated trust dynamic between human and robot teammates is especially critical to achieving effective collaboration. Trust-aware robot control frameworks leverage existing knowledge about trust in HRI to maintain optimal team performance during real-time, ongoing interaction. In this chapter we review the advantages and shortcomings of the two existing trust-aware control frameworks with respect to their use in the context of proximate human robot teaming. In the process we provide the foundation for the next generation of trust-aware, proximate interaction control frameworks, emphasizing the importance of robot movement behavior in shaping team trust dynamics.
Trust in autonomy is essential for effective human-robot collaboration and user adoption of autonomous systems such as robot assistants. This article introduces a computational model that integrates trust into robot decision making. Specifically, we learn from data a partially observable Markov decision process (POMDP) with human trust as a latent variable. The trust-POMDP model provides a principled approach for the robot to (i) infer the trust of a human teammate through interaction, (ii) reason about the effect of its own actions on human trust, and (iii) choose actions that maximize team performance over the long term. We validated the model through human subject experiments on a table clearing task in simulation (201 participants) and with a real robot (20 participants). In our studies, the robot builds human trust by manipulating low-risk objects first. Interestingly, the robot sometimes fails intentionally to modulate human trust and achieve the best team performance. These results show that the trust-POMDP calibrates trust to improve human-robot team performance over the long term. Further, they highlight that maximizing trust alone does not always lead to the best performance.
Objective
This research examined the effects of reliability and stated social intent on trust, trustworthiness, and one’s willingness to endorse use of an autonomous security robot (ASR).
Background
Human–robot interactions in the domain of security is plausible, yet we know very little about what drives acceptance of ASRs. Past research has used static images and game-based simulations to depict the robots versus actual humans interacting with actual robots.
Method
A video depicted an ASR interacting with a human. The ASR reviewed access credentials and allowed entrance once verified. If the ASR could not verify one’s credentials it instructed the visitor to return to the security checkpoint. The ASR was equipped with a nonlethal device and the robot used this device on one of the three visitors (a research confederate). Manipulations of reliability and stated social intent of the ASR were used in a 2 × 4 between subjects design ( N = 320).
Results
Reliability influenced trust and trustworthiness. Stated social intent influenced trustworthiness. Participants reported being more favorable toward use of the ASR in military contexts versus public contexts.
Conclusion
The study demonstrated that reliability of the ASR and statements regarding the ASR’s stated social intent are important considerations influencing the trust process (inclusive of intentions to be vulnerable and trustworthiness perceptions).
Application
If robotic systems are authorized to use force against a human, public acceptance may be increased with availability of the intent-based programming of the robot and whether or not the robot’s decision was reliable.
As intelligent systems gain autonomy and capability, it becomes vital to ensure that their objectives match those of their human users; this is known as the value-alignment problem. In robotics, value alignment is key to the design of collaborative robots that can integrate into human workflows, successfully inferring and adapting to their users’ objectives as they go. We argue that a meaningful solution to value alignment must combine multi-agent decision theory with rich mathematical models of human cognition, enabling robots to tap into people’s natural collaborative capabilities. We present a solution to the cooperative inverse reinforcement learning (CIRL) dynamic game based on well-established cognitive models of decision making and theory of mind. The solution captures a key reciprocity relation: the human will not plan her actions in isolation, but rather reason pedagogically about how the robot might learn from them; the robot, in turn, can anticipate this and interpret the human’s actions pragmatically. To our knowledge, this work constitutes the first formal analysis of value alignment grounded in empirically validated cognitive models.
Trust is essential in shaping human interactions with one another and with robots. In this article we investigate how human trust in robot capabilities transfers across multiple tasks. We present a human-subject study of two distinct task domains: a Fetch robot performing household tasks and a virtual reality simulation of an autonomous vehicle performing driving and parking maneuvers. The findings expand our understanding of trust and provide new predictive models of trust evolution and transfer via latent task representations: a rational Bayes model, a data-driven neural network model, and a hybrid model that combines the two. Experiments show that the proposed models outperform prevailing models when predicting trust over unseen tasks and users. These results suggest that (i) task-dependent functional trust models capture human trust in robot capabilities more accurately and (ii) trust transfer across tasks can be inferred to a good degree. The latter enables trust-mediated robot decision-making for fluent human–robot interaction in multi-task settings.
Workload management is of critical concern in the teleoperation of unmanned vehicles because teleoperation is often employed in high-risk industries wherein high workload can lead to sub-optimal task performance and can harm human operators’ long-term well-being. This study aimed to assess the detrimental effects of time delays in teleoperation on operators’ workload and performance, and how a delay compensation aid mitigated such effects. We conducted a human-in-the-loop experiment with 36 participants using a dual-task teleoperation platform, where participants drove a simulated High Mobility Multipurpose Wheeled Vehicle (HMMWV) and performed a one-back memory task under three conditions: the delay condition, the delay with compensation aid condition, and the ideal no delay condition. A model-free predictor was used as the compensation aid. Results indicate that with a time delay of 0.8-s participants’ workload increased and performance degraded significantly. Moreover, the model-free predictor mitigated the detrimental effects of time delay on workload and task performance. Our findings suggest that participants are more sensitive in their perceived workload compared to the objective and physiological measures of workload. In addition, without any delay compensation algorithms, continuous teleoperation may not be ideal for operations with long time delays.
In this paper, we establish a partially observable Markov decision process (POMDP) model framework that captures dynamic changes in human trust and workload for contexts that involve interactions between humans and intelligent decision-aid systems. We use a reconnaissance mission study to elicit a dynamic change in human trust and workload with respect to the system's reliability and user interface transparency as well as the presence or absence of danger. We use human subject data to estimate transition and observation probabilities of the POMDP model and analyze the trust-workload behavior of humans. Our results indicate that higher transparency is more likely to increase human trust when the existing trust is low but also is more likely to decrease trust when it is already high. Furthermore, we show that by using high transparency, the workload of the human is always likely to increase. In our companion paper, we use this estimated model to develop an optimal control policy that varies system transparency to affect human trust-workload behavior towards improving human-machine collaboration.
To attain improved human-machine collaboration, it is necessary for autonomous systems to infer human trust and workload and respond accordingly. In turn, autonomous systems require models that capture both human trust and workload dynamics. In a companion paper, we developed a trust-workload partially observable Markov decision process (POMDP) model framework that captured changes in human trust and workload for contexts that involve interaction between a human and an intelligent decision-aid system. In this paper, we define intuitive reward functions and show that these can be readily transformed for integration with the proposed POMDP model. We synthesize a near-optimal control policy using transparency as the feedback variable based on solutions for two cases: 1) increasing human trust and reducing workload, and 2) improving overall performance along with the aforementioned objectives for trust and workload. We implement these solutions in a reconnaissance mission study in which human subjects are aided by a virtual robotic assistant in completing a series of missions. We show that it is not always beneficial to aim to improve trust; instead, the control objective should be to optimize a context-specific performance objective when designing intelligent decision-aid systems that influence trust-workload behavior.
The current article reports the results of three different studies which explored the relationship between the Perfect Automation Schema (PAS) and trust in a human–machine context. It was expected that PAS would be related to higher trust and higher performance expectations for automation. Studies 1 and 2 used a human-robot interaction simulator as the context for human-machine interaction and study 3 used an applied sample of F-16 pilots and their views of the Automatic Ground Collision Avoidance System (Auto-GCAS) which is an advanced automated safety system. Results from all three studies demonstrated that the high expectations facet of PAS had a positive relationship with trust, but not the all-or-none facet. These results suggest that the PAS as measured by high expectations may be a fruitful construct for researchers in the domain of trust in automation.
Automation is often problematic because people fail to rely upon it appropriately. Because people respond to technology socially, trust influences reliance on automation. In particular, trust guides reliance when complexity and unanticipated situations make a complete understanding of the automation impractical. This review considers trust from the organizational, sociological, interpersonal, psychological, and neurological perspectives. It considers how the context, automation characteristics, and cognitive processes affect the appropriateness of trust. The context in which the automation is used influences automation performance and provides a goal-oriented perspective to assess automation characteristics along a dimension of attributional abstraction. These characteristics can influence trust through analytic, analogical, and affective processes. The challenges of extrapolating the concept of trust in people to trust in automation are discussed. A conceptual model integrates research regarding trust in automation and describes the dynamics of trust, the role of context, and the influence of display characteristics. Actual or potential applications of this research include improved designs of systems that require people to manage imperfect automation. Copyright © 2004, Human Factors and Ergonomics Society. All rights reserved.
Trust is essential for human-robot collaboration and user adoption of autonomous systems, such as robot assistants. This paper introduces a computational model which integrates trust into robot decision-making. Specifically, we learn from data a partially observable Markov decision process (POMDP) with human trust as a latent variable. The trust-POMDP model provides a principled approach for the robot to (i) infer the trust of a human teammate through interaction, (ii) reason about the effect of its own actions on human behaviors, and (iii) choose actions that maximize team performance over the long term. We validated the model through human subject experiments on a table-clearing task in simulation (201 participants) and with a real robot (20 participants). The results show that the trust-POMDP improves human-robot team performance in this task. They further suggest that maximizing trust in itself may not improve team performance.
Intuitively, obedience -- following the order that a human gives -- seems like a good property for a robot to have. But, we humans are not perfect and we may give orders that are not best aligned to our preferences. We show that when a human is not perfectly rational then a robot that tries to infer and act according to the human's underlying preferences can always perform better than a robot that simply follows the human's literal order. Thus, there is a tradeoff between the obedience of a robot and the value it can attain for its owner. We investigate how this tradeoff is impacted by the way the robot infers the human's preferences, showing that some methods err more on the side of obedience than others. We then analyze how performance degrades when the robot has a misspecified model of the features that the human cares about or the level of rationality of the human. Finally, we study how robots can start detecting such model misspecification. Overall, our work suggests that there might be a middle ground in which robots intelligently decide when to obey human orders, but err on the side of obedience.
Intuitively, obedience -- following the order that a human gives -- seems like a good property for a robot to have. But, we humans are not perfect and we may give orders that are not best aligned to our preferences. We show that when a human is not perfectly rational then a robot that tries to infer and act according to the human's underlying preferences can always perform better than a robot that simply follows the human's literal order. Thus, there is a tradeoff between the obedience of a robot and the value it can attain for its owner. We investigate how this tradeoff is impacted by the way the robot infers the human's preferences, showing that some methods err more on the side of obedience than others. We then analyze how performance degrades when the robot has a misspecified model of the features that the human cares about or the level of rationality of the human. Finally, we study how robots can start detecting such model misspecification. Overall, our work suggests that there might be a middle ground in which robots intelligently decide when to obey human orders, but err on the side of obedience.