No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Conversate: Supporting Reflective Learning in Interview Practice Through Interactive Simulation and Dialogic Feedback
Abstract
Job interviews play a critical role in shaping one's career, yet practicing interview skills can be challenging, especially without access to human coaches or peers for feedback. Recent advancements in large language models (LLMs) present an opportunity to enhance the interview practice experience. Yet, little research has explored the effectiveness and user perceptions of such systems or the benefits and challenges of using LLMs for interview practice. Furthermore, while prior work and recent commercial tools have demonstrated the potential of AI to assist with interview practice, they often deliver one-way feedback, where users only receive information about their performance. By contrast, dialogic feedback , a concept developed in learning sciences, is a two-way interaction feedback process that allows users to further engage with and learn from the provided feedback through interactive dialogue. This paper introduces Conversate, a web-based application that supports reflective learning in job interview practice by leveraging large language models (LLMs) for interactive interview simulations and dialogic feedback. To start the interview session, the user provides the title of a job position (e.g., entry-level software engineer) in the system. Then, our system will initialize the LLM agent to start the interview simulation by asking the user an opening interview question and following up with questions carefully adapted to subsequent user responses. After the interview session, our back-end LLM framework will then analyze the user's responses and highlight areas for improvement. Users can then annotate the transcript by selecting specific sections and writing self-reflections. Finally, the user can interact with the system for dialogic feedback, conversing with the LLM agent to learn from and iteratively refine their answers based on the agent's guidance. To evaluate Conversate, we conducted a user study with 19 participants to understand their perceptions of using LLM-supported interview simulation and dialogic feedback. Our findings show that participants valued the adaptive follow-up questions from LLMs, as they enhanced the realism of interview simulations and encouraged deeper thinking. Participants also appreciated the AI-assisted annotation, as it reduced their cognitive burden and mitigated excessive self-criticism in their own evaluation of their interview performance. Moreover, participants found the LLM-supported dialogic feedback to be beneficial, as it promoted personalized and continuous learning, reduced feelings of judgment, and allowed them to express disagreement.
... Artificial intelligence (AI) has gained significant interest in automating team tasks and interactions among team members [83]. In particular, AI-generated feedback can help organizations automate performance evaluation, streamline team monitoring, and provide continuous improvement suggestions without the need for constant human intervention [8,23,73]. Moreover, with the increasing number of teams in organizations and institutions, automating feedback can become an important tool to address scalability issues. ...
Providing timely and actionable feedback is crucial for effective collaboration, learning, and coordination within teams. However, many teams face challenges in receiving feedback that aligns with their goals and promotes cohesion. We introduce tAIfa (``Team AI Feedback Assistant''), an AI agent that uses Large Language Models (LLMs) to provide personalized, automated feedback to teams and their members. tAIfa analyzes team interactions, identifies strengths and areas for improvement, and delivers targeted feedback based on communication patterns. We conducted a between-subjects study with 18 teams testing whether using tAIfa impacted their teamwork. Our findings show that tAIfa improved communication and contributions within the teams. This paper contributes to the Human-AI Interaction literature by presenting a computational framework that integrates LLMs to provide automated feedback, introducing tAIfa as a tool to enhance team engagement and cohesion, and providing insights into future AI applications to support team collaboration.
... It is also the case that the amount of such clarificatory information can often be surprisingly small [5]. It is, therefore, the case that LLMs have the mechanisms necessary for conducting interactive conversations [6]. But, a question of some conceptual importance is this: when do LLMs need interactive feedback? ...
Natural language as a medium for human-computer interaction has long been anticipated, has been undergoing a sea-change with the advent of Large Language Models (LLMs) with startling capacities for processing and generating language. Many of us now treat LLMs as modern-day oracles, asking it almost any kind of question. Unlike its Delphic predecessor, consulting an LLM does not have to be a single-turn activity (ask a question, receive an answer, leave); and -- also unlike the Pythia -- it is widely acknowledged that answers from LLMs can be improved with additional context. In this paper, we aim to study when we need multi-turn interactions with LLMs to successfully get a question answered; or conclude that a question is unanswerable. We present a neural symbolic framework that models the interactions between human and LLM agents. Through the proposed framework, we define incompleteness and ambiguity in the questions as properties deducible from the messages exchanged in the interaction, and provide results from benchmark problems, in which the answer-correctness is shown to depend on whether or not questions demonstrate the presence of incompleteness or ambiguity (according to the properties we identify). Our results show multi-turn interactions are usually required for datasets which have a high proportion of incompleteness or ambiguous questions; and that that increasing interaction length has the effect of reducing incompleteness or ambiguity. The results also suggest that our measures of incompleteness and ambiguity can be useful tools for characterising interactions with an LLM on question-answeringproblems
Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate , where to evaluate , and how to evaluate . Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, education, natural and social sciences, agent applications, and other areas. Secondly, we answer the ‘where’ and ‘how’ questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing the performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey.
The question of how generative AI tools, such as large language models and chatbots, can be leveraged ethically and effectively in education is ongoing. Given the critical role that writing plays in learning and assessment within educational institutions, it is of growing importance for educators to make thoughtful and informed decisions as to how and in what capacity generative AI tools should be leveraged to assist in the development of students' writing skills. This paper reports on two longitudinal studies. Study 1 examined learning outcomes of 48 university English as a new language (ENL) learners in a six-week long repeated measures quasi experimental design where the experimental group received writing feedback generated from Chat-GPT (GPT-4) and the control group received feedback from their human tutor. Study 2 analyzed the perceptions of a different group of 43 ENLs who received feedback from both ChatGPT and their tutor. Results of study 1 showed no difference in learning outcomes between the two groups. Study 2 results revealed a near even split in preference for AI-generated or human-generated feedback, with clear advantages to both forms of feedback apparent from the data. The main implication of these studies is that the use of AI-generated feedback can likely be incorporated into ENL essay evaluation without affecting learning outcomes, although we recommend a blended approach that utilizes the strengths of both forms of feedback. The main contribution of this paper is in addressing generative AI as an automatic essay evaluator while incorporating learner perspectives.
Theories and research in human–machine communication (HMC) suggest that machines, when replacing humans as communication partners, change the processes and outcomes of communication. With artificial intelligence (AI) increasingly used to interview and evaluate job applicants, employers should consider the effects of AI on applicants’ psychology and performance during AI-based interviews. This study examined job applicants’ experience and speech fluency when evaluated by AI. In a three-condition between-subjects experiment (N = 134), college students had an online mock job interview under the impression that their performance would be evaluated by a human recruiter, an AI system, or an AI system with a humanlike interface. Participants reported higher uncertainty and lower social presence and had a higher articulation rate in the AI-evaluation condition than in the human-evaluation condition. Through lowering social presence, AI evaluation increased speech rate and reduced silent pauses. Findings inform theories of HMC and practices of automated recruitment and professional training.
The user experience of an asynchronous video interview system, conventionally is not reciprocal or conversational. Interview applicants expect that, like a typical face-to-face interview, they are innate and coherent. We posit that the planned adoption of limited probing through follow-up questions is an important step towards improving the interaction. We propose a follow-up question generation model (followQG) capable of generating relevant and diverse follow-up questions based on the previously asked questions, and their answers. We implement a 3D virtual interviewing system, Maya, with capability of follow-up question generation. Existing asynchronous interviewing systems are not dynamic with scripted and repetitive questions. In comparison, Maya responds with relevant follow-up questions, a largely unexplored feature of irtual interview systems. We take advantage of the implicit knowledge from deep pre-trained language models to generate rich and varied natural language follow-up questions. Empirical results suggest that followQG generates questions that humans rate as high quality, achieving 77% relevance. A comparison with strong baselines of neural network and rule-based systems show that it produces better quality questions. The corpus used for fine-tuning is made publicly available.
Large language models represent a significant advancement in the field of AI. The underlying technology is key to further innovations and, despite critical views and even bans within communities and regions, large language models are here to stay. This commentary presents the potential benefits and challenges of educational applications of large language models, from student and teacher perspectives. We briefly discuss the current state of large language models and their applications. We then highlight how these models can be used to create educational content, improve student engagement and interaction, and personalize learning experiences. With regard to challenges, we argue that large language models in education require teachers and learners to develop sets of competencies and literacies necessary to both understand the technology as well as their limitations and unexpected brittleness of such systems. In addition, a clear strategy within educational systems and a clear pedagogical approach with a strong focus on critical thinking and strategies for fact checking are required to integrate and take full advantage of large language models in learning settings and teaching curricula. Other challenges such as the potential bias in the output, the need for continuous human oversight, and the potential for misuse are not unique to the application of AI in education. But we believe that, if handled sensibly, these challenges can offer insights and opportunities in education scenarios to acquaint students early on with potential societal biases, criticalities, and risks of AI applications. We conclude with recommendations for how to address these challenges and ensure that such models are used in a responsible and ethical manner in education.
Personal tracking devices are equipped with more and more sensors and offer an ever-increasing level of accuracy. Yet, this comes at the cost of increased complexity. To deal with that problem, fitness trackers use derived metrics---scores calculated based on sensor data, e.g. a stress score. This means that part of the agency in interpreting health data is transferred from the user to the tracker. In this paper, we investigate the consequences of that transition and study how derived metrics can be designed to offer an optimal personal informatics experience. We conducted an online survey and a series of interviews which examined a health score (a hypothetical derived metric) at three levels of abstraction. We found that the medium abstraction level led to the highest level of reflection. Further, we determined that presenting the metric without contextual information led to decreased transparency and meaning. Our work contributes guidelines for designing effective derived metrics.
Despite the wide implementation of self-reflection in higher education, the body of literature has predominantly focused on students from the clinical health sciences rather than public health. The objective of this study was to evaluate the correlates as well as to explore the motivating factors and barriers of engaging in regular self-reflection among public health students in higher education. We used a mixed methods approach (explanatory sequential design), comprising a cross-sectional survey (quantitative phase) followed by in-depth interviews (qualitative phase). We evaluated the association between reflection frequency as well as the perceptions and facilitators in reflection using the modified Breslow-Cox proportional hazards regression model. Prevalence ratio (PR) was generated. Thematic data analysis was carried out to analyse the qualitative data. Quantitative findings revealed being a regular reflector was positively associated with being more motivated to learn when one applied self-reflection (adjusted PR 1.60, 95% CI 1.17–2.20), the perception of being more prepared for a public health career in the future (adjusted PR 1.28, 95% CI 1.02–1.60), as well as being given enough opportunities to carry out self-reflection in the public health modules (adjusted PR 1.24, 95% CI 1.05–1.45). Qualitative findings revealed most students started their self-reflection mainly due to extrinsic factors such as institutional support, social support, teacher influence and environmental influence. Of these, the most prominent was teacher influence, indicating that they are key agents in promoting self-reflection. Students expressed that it would be important to cultivate intrinsic motivation to sustain their practice of self-reflection along the learning journey such as for the development of career-related professional skills. Other than intrinsic motivation, environmental influences were also important to promote continual reflection among students such as the availability of ample opportunities. Prominent barriers to address included external student factors such as the imbalanced power relationship between teacher and student, and internal student factors such as the perception that self-reflection was too cumbersome and time consuming.
Feedback is information provided to recipients about their behavior, performance, or understanding, the goal of which is to foster recipients’ self-awareness, and behavioral reinforcement or change. Yet, feedback often fails to achieve this goal. For feedback to be effective, recipients must be receptive and accurately understand the meaning and veracity of the feedback (i.e., discern the truth in feedback). Honesty is critically important for both receptivity and discerning the truth in the feedback. In this article, we identify barriers to receptivity and discerning the truth in feedback and illustrate how these barriers hinder recipients’ learning and improvement. Barriers can arise from the feedback itself, the feedback-giver, and the feedback-recipient, and both parties share responsibility for removing them.
e core focus of this review is to show how immediate and valid feedback, qualitative assessment in uence enhances students learning in a higher education environment. With the rising trend of online education especially in this COVID-19 pandemic, the role of assessment and feedback also changes. Earlier the assessment part is not considered the main focus in learning and teaching in HEIs, but now with the increase in online education, it is observed that the paradigm is shifted toward assessing those activities of students that enhance their learning outcomes. A lot of research work has been done on developing assessment strategies and techniques that can support learning and teaching e ectively. Yet, there is limited research that looks at how methods applied in learning analytics can be used and possibly constitutes the assessment process. e objective of this work is to provide an exploratory and comparative study of how assessment and feedback practices can enhance students learning outcomes using AI. e key contribution of this study attempts to capture an outline of the most used arti cial intelligence and machine learning algorithms for student success. e results showed that I-FCN performed better than other techniques (ANN, XG Boost, SVM, Random Forest, and Decision Trees) in all measured performance metrics. Also, the result of the comparative analysis study will help the educators, instructors, and administrators on how they could take the advantage of a data-driven approach, design less pressurized, more valid, reliable, constructive assessment ndings, and connect the power of assessment and feedback to enhance the learning outcomes.
Reflection is a commonly addressed design goal in commercial systems and in Human-Computer Interaction (HCI) research. Yet, it is still unclear what tools are at the disposal of designers who want to build systems that support reflection. Understanding the design space of reflection support systems and the interaction techniques that can foster reflection is necessary to enable building technologies that contribute to the users' well-being. In order to gain additional insight into how interactive artefacts foster reflection, we investigated past research prototypes and reflection-supporting smartphone applications (apps). Through a structured literature review and an analysis of app reviews, we constructed four design resources for reflection: temporal perspective, conversation, comparison and discovery. We also identified design patterns in past digital artefacts that implement the resources. Our work constitutes intermediate-level knowledge that is intended to inspire future technologies that better support reflection.
This work discusses a nudging intervention mechanism combined with an artificial intelligence (AI) system for early detection of learners’ risk of failing or dropping out. Different types of personalized nudges were designed according to educational principles and the learners’ risk classification. The impact on learners’ performance, dropout reduction, and satisfaction was evaluated through a study with 252 learners in a first-year course at a fully online university. Different learners’ groups were designed, with each receiving a different set of nudges. Results showed that nudges positively impacted the learners’ performance and satisfaction, and reduced dropout rates. The impact significantly increased when different types of nudges were provided. Our research reinforced the role of AI as useful in online, distance, and open learning for providing timely learner support, improved learning experiences, and enhanced learner-teacher communication.
Preparing students to collaborate with AI remains a challenging goal. As AI technologies are new to K-12 schools, there is a lack of studies that inform how to design learning when AI is introduced as a collaborative learning agent to classrooms. The present study, therefore, aimed to explore teachers’ perspectives on what (1) curriculum design, (2) student-AI interaction, and (3) learning environments are required to design student-AI collaboration (SAC) in learning and (4) how SAC would evolve. Through in-depth interviews with 10 Korean leading teachers in AI in Education (AIED), the study found that teachers perceived capacity and subject-matter knowledge building as the optimal learning goals for SAC. SAC can be facilitated through interdisciplinary learning, authentic problem solving, and creative tasks in tandem with process-oriented assessment and collaboration performance assessment. While teachers expressed instruction on AI principles, data literacy, error analysis, AI ethics, and AI experiences in daily life were crucial support, AI needs to offer an instructional scaffolding and possess attributes as a learning mate to enhance student-AI interaction. In addition, teachers highlighted systematic AIED policy, flexible school system, the culture of collaborative learning, and a safe to fail environment are significant. Teachers further anticipated students would develop collaboration with AI through three stages: (1) learn about AI, (2) learn from AI, and (3) learn together. These findings can provide a more holistic understanding of the AIED and implications for the educational policies, educational AI design as well as instructional design that are aimed at enhancing SAC in learning.
An emerging research agenda in Computer-Supported Cooperative Work focuses on human-agent teaming and AI agent's roles and effects in modern teamwork. In particular, one understudied key question centers around the construct of team cognition within human-agent teams. This study explores the unique nature of team dynamics in human-agent teams compared to human-human teams and the impact of team composition on perceived team cognition, team performance, and trust. In doing so, a mixed-method approach, including three team composition conditions (all human, human-human-agent, human-agent-agent), completed the team simulation NeoCITIES and completed shared mental model, trust, and perception measures. Results found that human-agent teams are similar to human-only teams in the iterative development of team cognition and the importance of communication to accelerating its development; however, human-agent teams are different in that action-related communication and explicitly shared goals are beneficial to developing team cognition. Additionally, human-agent teams trusted agent teammates less when working with only agents and no other humans, perceived less team cognition with agent teammates than human ones, and had significantly inconsistent levels of team mental model similarity when compared to human-only teams. This study contributes to Computer-Supported Cooperative Work in three significant ways: 1) advancing the existing research on human-agent teaming by shedding light on the relationship between humans and agents operating in collaborative environments, 2) characterizing team cognition development in human-agent teams; and 3) advancing real-world design recommendations that promote human-centered teaming agents and better integrate the two.
In the past few decades, technology has completely transformed the world around us. Indeed, experts believe that the next big digital transformation in how we live, communicate, work, trade and learn will be driven by Artificial Intelligence (AI) [83]. This paper presents a high-level industrial and academic overview of AI in Education (AIEd). It presents the focus of latest research in AIEd on reducing teachers’ workload, contextualized learning for students, revolutionizing assessments and developments in intelligent tutoring systems. It also discusses the ethical dimension of AIEd and the potential impact of the Covid-19 pandemic on the future of AIEd’s research and practice. The intended readership of this article is policy makers and institutional leaders who are looking for an introductory state of play in AIEd.
With the development of computing and information processing techniques, artificial intelligence (AI) has been extensively applied in education. Artificial intelligence in education (AIEd) opens new opportunities, potentials, and challenges in educational practices. In its short history, AIEd has been undergoing several paradigmatic shifts, which are characterized into three paradigms in this position paper: AI-directed, learner-as-recipient, AI-supported, learner-as-collaborator, and AI-empowered, learner-as-leader. In three paradigms, AI techniques are used to address educational and learning issues in varied ways. AI is used to represent knowledge models and direct cognitive learning while learners are recipients of AI service in Paradigm One; AI is used to support learning while learners work as collaborators with AI in Paradigm Two; AI is used to empower learning while learners take agency to learn in Paradigm Three. Overall, the development trend of AIEd has been developing to empower learner agency and personalization, enable learners to reflect on learning and inform AI systems to adapt accordingly, and lead to an iterative development of the learner-centered, data-driven, personalized learning.
Video recording is one of the most commonly used techniques for reflection, because video allows people to know what they look like to others and how they could improve their performance, but it is problematic because some people easily fall into negative emotions and worry about their performance, resulting in a low benefit. In this study, the possibility of applying a simple VR-based reflection method was explored. This method uses virtual reality (VR) and a head-mounted display (HMD) to allow presenters to watch their own presentations from the audience’s perspective and uses an avatar, which hides personal appearance, which has low relevance to the quality of presentation, to help reduce self-awareness during reflection. An experimental study was carried out, considering four personal characteristics—gender, personal anxiety, personal confidence and self-bias. The goal of this study is to discuss which populations can benefit more from this system and to assess the impact of the avatar and HMD-based VR. According to the results, the individuals with low self-confidence in their public speaking skills could benefit more on self-evaluation from VR reflection with HMD, while individuals with negative self-bias could reduce more anxiety by using an avatar.
Video-Reflection is a common approach to realize reflection in the field of executive coaching for professional development, which presents a video recording of the coaching session to a coachee in order to make the coachee reflectively think about oneself. However, it requires a great deal of time to watch the full length of the video and is highly dependent on the skills of the coach. We expect that the quality and efficiency of video-reflection can be improved with the support of computers. In this paper, we introduce INWARD, a computational tool that leverages human behavior analysis and video-based interaction techniques. The results of a user study involving 20 coaching sessions with five coaches indicate that INWARD enables efficient video-reflection and, by leveraging meta-reflection, realizes the ameliorated outcome of executive coaching. Moreover, discussions based on comments from the participants support the effectiveness of INWARD and suggest further possibilities of computer-supported approaches.
Self confidence, self esteem and self efficacy are widely common terms in job search effectiveness according to literature. There is a consensus in a large extent that candidates’ self confidence during interview process has significant influence and affects recruiters’ decisions and respectively hiring results. This study aims to search the significance of employee candidates’ self confidence and its relative impact in recruiters, employers, hiring decisions during employment interview. For this purpose, an empirical research has been conducted in order to explore the importance and relative impact of candidates’ self-confidence in a sample of two hundred and sixty recruiters’, personnel managers’ and employers in Greece about the impact of candidates’ self confidence, might have in their hiring decisions, during interview selection process. The significance of candidates self confidence was measured according to three specific dimensions of the recruiters decisions, which were in particular, the possibility of hiring them, the possibility of consider them as suitable for the position they applied for, and thirdly for the possibility of giving them a chance for a second interview. The three measures were analysed each one individually, and then unified in one, with the term hireability- sellectability. The new index has been tested for cronbach’s alpha measure of reliability, taking high acceptant value. Results show that employee candidates’ self confidence has significant influence and affect recruiters’ decisions and selection outcomes regarding this index. There is also significant impact of candidate’s self confidence, in all three of interviewers’ decisions options as in particular, to hire the candidates, to consider them suitable for the position and to invite them in a second interview.
The typical hiring pipeline for software engineering occurs over several stages-from phone screening and technical on-site interviews, to offer and negotiation. When these hiring pipelines are "leaky," otherwise qualified candidates are lost at some stage of the pipeline. These leaky pipelines impact companies in several ways, including hindering a company's ability to recruit competitive candidates and build diverse software teams. To understand where candidates become disengaged in the hiring pipeline-and what companies can do to prevent it-we conducted a qualitative study on over 10,000 reviews on 19 companies from Glassdoor, a website where candidates can leave reviews about their hiring process experiences. We identified several poor practices which prematurely sabotage the hiring process-for example, not adequately communicating hiring criteria, conducting interviews with inexperienced interviewers, and ghosting candidates. Our findings provide a set of guidelines to help companies improve their hiring pipeline practices-such as being deliberate about phrasing and language during initial contact with the candidate, providing candidates with constructive feedback after their interviews, and bringing salary transparency and long-term career discussions into offers and negotiations. Operationalizing these guidelines helps make the hiring pipeline more transparent, fair, and inclusive.
Automatic assessment of soft skills is an interesting problem in social computing. Soft skills are essential to any individual for personal and career development. Soft skills assessment includes Big Five personality, social and communication skill and leadership skill can now be enabled using behavior tracking and mapping behavior to perception. Such assessments can be used for manpower selection and training. This paper reviews the existing literature on automatic analysis of social interactions for soft skill assessment in different contexts. A variety of social situations offline and online methods are considered for assessment. Offline methods include traditional forms of discussion without any technology mediation. Online methods utilize technology for scalability and alleviate the assessment process without collocation of multiple parties. We address some of the challenges that are still open in the field.
Assessment feedback gains consistently low satisfaction scores in national surveys of student satisfaction, with most concern surrounding its timeliness, quality and effectiveness. We present the results of a two year qualitative study, thematically analysing semi-structured interviews with students who have undertaken dialogic feed-forward coursework on a second year undergraduate geography module in a British university. The assessment consists of submitting a considered draft of a coursework essay, which is discussed and evaluated face-to-face with the course tutor before a self-reflective piece is written about the assessment process and a final essay is submitted for formal grading. We present evidence that this process asserts a positive influence on the student learning experience in a number of inter-related cognitive and affective ways, impacting upon learning behaviour, supporting student achievement, and raising NSS scores related to feedback. We espouse an ipsative, cyclical approach to dialogic feed-forward, focusing on learners’ longitudinal development.
A common barrier to healthcare for psychiatric conditions is the stigma associated with these disorders. Perceived stigma prevents many from reporting their symptoms. Stigma is a particularly pervasive problem among military service members, preventing them from reporting symptoms of combat-related conditions like posttraumatic stress disorder (PTSD). However, research shows (increased reporting by service members when anonymous assessments are used. For example, service members report more symptoms of PTSD when they anonymously answer the Post-Deployment Health Assessment (PDHA) symptom checklist compared to the official PDHA, which is identifiable and linked to their military records. To investigate the factors that influence reporting of psychological symptoms by service members, we used a transformative technology: automated virtual humans that interview people about their symptoms. Such virtual human interviewers allow simultaneous use of two techniques for eliciting disclosure that would otherwise be incompatible; they afford anonymity while also building rapport. We examined whether virtual human interviewers could increase disclosure of mental health symptoms among active-duty service members that just returned from a year-long deployment in Afghanistan. Service members reported more symptoms during a conversation with a virtual human interviewer than on the official PDHA. They also reported more to a virtual human interviewer than on an anonymized PDHA. A second, larger sample of active-duty and former service members found a similar effect that approached statistical significance. Because respondents in both studies shared more with virtual human interviewers than an anonymized PDHA—even though both conditions control for stigma and ramifications for service members’ military records—virtual human interviewers that build rapport may provide a superior option to encourage reporting.
In the context of a Transition to Work course for fourth-year psychology majors, we had students use an e-portfolio to self-reflect on the learning experiences they deemed most significant during their degree. Such significant learning experiences can be drawn upon when answering behavioral job interview questions. We examined whether students would show improvement in mock behavioral interview performance over time, and whether any potential improvement was related to performance on self-reflective course assignments. Students indicated that the majority of their significant learning experiences had taken place outside of the classroom (e.g., paid and unpaid work, extracurricular activities). Across the duration of the course, students improved on all metrics of interview performance, and final interview performance was predicted by student grades on a self-reflective e-portfolio, but not other course assignments. Copyright © 2017 The International Society for the Scholarship of Teaching and Learning
Purpose
The purpose of this study was to investigate the association between leader arrogance on subordinate outcomes of feedback seeking, morale, and burnout through its relationships with subordinate feedback environment perceptions. Additionally, perceived organizational support and subordinate feedback orientation are examined as moderators that influence the degree to which leader arrogance exerts its effects on these outcomes.
Design/Methodology/Approach
Survey data were obtained from 302 participants on Amazon’s Mechanical Turk website and analyzed using Hayes’ (2013) PROCESS in SPSS.
Findings
Subordinates with more arrogant supervisors reported less favorable feedback environment perceptions, and subsequently, lower levels of feedback seeking, morale, and higher levels of burnout. Perceived organizational support and feedback orientation were identified as significant moderators in these relationships. Subordinates were less vulnerable to the negative outcomes of leader arrogance when they experienced higher levels of perceived organizational support. Finally, subordinates with favorable feedback orientations exhibited lower levels of feedback seeking in the face of the unfavorable feedback environments associated with arrogant leaders.
Implications
Given these findings, leader arrogance should be of great concern to organizations, as subordinates exposed to arrogant leaders are likely to experience adverse outcomes. Supplementing perceptions of organizational support may help alleviate some of these effects. Additionally, subordinates with favorable feedback orientations may be particularly vulnerable to the effects of leader arrogance on outcomes of feedback seeking and morale.
Originality/Value
This study is the first to demonstrate the interpersonal implications of leader arrogance for subordinates, as well as explore mediators that play a role in these relationships.
The emergence of large language models (LLMs) has marked a significant breakthrough in natural language processing (NLP), fueling a paradigm shift in information acquisition. Nevertheless, LLMs are prone to hallucination, generating plausible yet nonfactual content. This phenomenon raises significant concerns over the reliability of LLMs in real-world information retrieval (IR) systems and has attracted intensive research to detect and mitigate such hallucinations. Given the open-ended general-purpose attributes inherent to LLMs, LLM hallucinations present distinct challenges that diverge from prior task-specific models. This divergence highlights the urgency for a nuanced understanding and comprehensive overview of recent advances in LLM hallucinations. In this survey, we begin with an innovative taxonomy of hallucination in the era of LLM and then delve into the factors contributing to hallucinations. Subsequently, we present a thorough overview of hallucination detection methods and benchmarks. Our discussion then transfers to representative methodologies for mitigating LLM hallucinations. Additionally, we delve into the current limitations faced by retrieval-augmented LLMs in combating hallucinations, offering insights for developing more robust IR systems. Finally, we highlight the promising research directions on LLM hallucinations, including hallucination in large vision-language models and understanding of knowledge boundaries in LLM hallucinations.
Job interviews are the most widely accepted method for companies to select suitable candidates, and a critical challenge is finding the right questions to ask job candidates. Moreover, there is a lack of integrated tools for automatically generating interview questions and recommending the right questions to interviewers. To this end, in this paper, we propose an intelligent system for assisting job interviews, namely, DuerQues. To build this system, we first investigate how to automatically generate skill-oriented interview questions in a scalable way by learning external knowledge from online knowledge-sharing communities. Along this line, we develop a novel distantly supervised skill entity recognition method to identify skill entities from large-scale search queries and web page titles with less need for human annotation. Additionally, we propose a neural generative model for generating skill-oriented interview questions. In particular, we introduce a data-driven solution to create high-quality training instances and design a learning algorithm to improve the performance of question generation. Furthermore, we exploit click-through data from query logs and design a recommender system for recommending suitable questions to interviewers. Specifically, we introduce a graph-enhanced algorithm to efficiently recommend suitable questions given a set of queried skills. Finally, extensive experiments on real-world datasets demonstrate the effectiveness of our DuerQues system in terms of the quality of generated skill-oriented questions and the performance of question recommendation.
Computational techniques offer a means to overcome the amplified complexity and resource-intensity of qualitative research on online communities. However, we lack an understanding of how these techniques are integrated by researchers in practice, and how to address concerns about researcher agency in the qualitative research process. To explore this gap, we deployed the Computational Thematic Analysis Toolkit to a team of public health researchers, and compared their analysis to a team working with traditional tools and methods. Each team independently conducted a thematic analysis of a corpus of comments from Canadian news sites to understand discourses around vaccine hesitancy. We then compared the analyses to investigate how computational techniques may have influenced their research process and outcomes. We found that the toolkit provided access to advanced computational techniques for researchers without programming expertise, facilitated their interaction and interpretation of the data, but also found that it influenced how they approached their thematic analysis.
Driven by the need to provide continuous, timely, and efficient customer service, firms are constantly experimenting with emerging technological solutions. In recent times firms have shown an increased interest in designing and implementing artificial intelligence (AI)-based interactional technologies, such as conversational AI agents and chatbots, that obviate the need for having human service agents for the provision of customer service. However, the business impact of conversational AI is contingent on customers using and adequately engaging with these tools. This engagement depends, in turn, on conversational AI’s similarity, or likeness to the human beings it is intended to replace. Businesses therefore need to understand what human-like characteristics and competencies should be embedded in customer-facing conversational AI agents to facilitate smooth user interaction. This focus on “human-likeness” for facilitating user engagement in the case of conversational AI agents is in sharp contrast to most prior information systems (IS) user engagement research, which is predicated on the “instrumental value” of information technology (IT). Grounding our work in the individual human competency and media naturalness literatures, we theorize the key role of human-like interactional competencies in conversational AI agents—specifically, cognitive, relational, and emotional competencies—in facilitating user engagement. We also hypothesize the mediating role of user trust in these relationships. Following a sequential mixed methods approach, we use a quantitative two-wave, survey-based study to test our model. We then examine the results in light of findings from qualitative follow-up interviews with a sampled set of conversational AI users. Together, the results offer a nuanced understanding of desirable human-like competencies in conversational AI agents and the salient role of user trust in fostering user engagement with them. We also discuss the implications of our study for research and practice.
We provide a philosophical explanation of the relation between artificial intelligence (AI) explainability and trust in AI, providing a case for expressions, such as “explainability fosters trust in AI,” that commonly appear in the literature. This explanation relates the justification of the trustworthiness of an AI with the need to monitor it during its use. We discuss the latter by referencing an account of trust, called “trust as anti-monitoring,” that different authors contributed developing. We focus our analysis on the case of medical AI systems, noting that our proposal is compatible with internalist and externalist justifications of trustworthiness of medical AI and recent accounts of warranted contractual trust. We propose that “explainability fosters trust in AI” if and only if it fosters justified and warranted paradigmatic trust in AI, i.e., trust in the presence of the justified belief that the AI is trustworthy, which, in turn, causally contributes to rely on the AI in the absence of monitoring. We argue that our proposed approach can intercept the complexity of the interactions between physicians and medical AI systems in clinical practice, as it can distinguish between cases where humans hold different beliefs on the trustworthiness of the medical AI and exercise varying degrees of monitoring on them. Finally, we apply our account to user’s trust in AI, where, we argue, explainability does not contribute to trust. By contrast, when considering public trust in AI as used by a human, we argue, it is possible for explainability to contribute to trust. Our account can explain the apparent paradox that in order to trust AI, we must trust AI users not to trust AI completely. Summing up, we can explain how explainability contributes to justified trust in AI, without leaving a reliabilist framework, but only by redefining the trusted entity as an AI-user dyad.
Assessment feedback gains consistently low satisfaction scores in national surveys of student satisfaction, with most concern surrounding its timeliness, quality and effectiveness. We present the results of a two year qualitative study, thematically analysing semi-structured interviews with students who have undertaken dialogic feed-forward coursework on a second year undergraduate geography module in a British university. The assessment consists of submitting a considered draft of a coursework essay, which is discussed and evaluated face-to-face with the course tutor before a self-reflective piece is written about the assessment process and a final essay is submitted for formal grading. We present evidence that this process asserts a positive influence on the student learning experience in a number of inter-related cognitive and affective ways, impacting upon learning behaviour, supporting student achievement, and raising NSS scores related to feedback. We espouse an ipsative, cyclical approach to dialogic feed-forward, focusing on learners’ longitudinal development.
This article reports on the findings of a study that investigated the effects of instructional conditions and prior experience on students’ self-reflection. The study was conducted with the use of a video annotation tool that was used by undergraduate performing arts students to reflect on their video-recorded performances. The study shows a consistent positive effect of previous experience with the video annotation tool for engagement with reflection. Graded instructional conditions with feedback had a positive effect on increasing higher order reflections particularly for students with prior experience with the video annotation tool for reflective purposes. The finding suggests that when including reflection in the curriculum, it is important to consider introducing it at a program or degree level rather than individual courses in order to provide an opportunity for students to gain experience with reflection and any particular tool that is used (e.g., a video annotation tool). Furthermore, reflective tasks should be scaffolded into the curriculum with ample opportunity for formative feedback and summative assessment in order to encourage higher order thinking and foster students’ metacognitive awareness and monitoring for increased goal-setting and acknowledgement of the motive or effect of their observed behavior.