Brianna J. Tomlinson’s research while affiliated with Georgia Institute of Technology and other places

A key psychological component of interactions in both human-human and human-automation relationships is trust. Although trust has repeatedly been conceptualized as having a component of risk, the role risk plays, as well as what elements of risk impact trust (e.g., perceived risk, risk-taking propensity), has not been clearly explained. Upon reviewing the foundational theories of trust, it is clear that trust is only needed when risk exists or is perceived to exist, in both human-human and human-automation contexts. Within the limited research that has explored human-automation trust and risk, it has been found that the presence of risk and a participant’s perceived situational risk impacts their behavioural trust of the automation. In addition, perceived relational risk has a strong negative relationship with trust. We provide an enhanced model of trust to demonstrate how risk interacts with trust, incorporating these distinct perceived risks, as well as risk-taking propensity. This model identifies the unique interactions of these components with trust based on both the theory reviewed and the studies that have explored some aspects of these relationships. Guidelines are provided for improving the study of human-automation trust via the incorporation of risk.

Interactive simulations are tools that can help students understand and learn about complex relationships. While most simulations are primarily visual due to mostly historical reasons, sounds can be used to add to the experience. In this work, we evaluated sets of audio designs for two different, but contextually- and visually-similar simulations. We identified key aspects of the audio representations and the simulation content which needed to be evaluated, and compared designs across two simulations to understand which auditory designs could generalize to other simulations. To compare the designs and explore how audio affected a user’s experience, we measured preference (through usability, user experience, and open-ended questions) and interpretation accuracy for different aspects of the simulation (including the main relationships and control feedback). We suggest important characteristics to represent through audio for future simulations, provide sound design suggestions, and address how overlap between visual and audio representations can support learning opportunities.

For persons with visual impairment, forming cognitive maps of unfamiliar interior spaces can be challenging. Various technical developments have converged to make it feasible, without specialized equipment, to represent a variety of useful landmark objects via spatial audio, rather than solely dispensing route information. Although such systems could be key to facilitating cognitive map formation, high-density auditory environments must be crafted carefully to avoid overloading the listener. This article recounts a set of research exercises with potential users, in which the optimization of such systems was explored. In Experiment 1, a virtual reality environment was used to rapidly prototype and adjust the auditory environment in response to participant comments. In Experiment 2, three variants of the system were evaluated in terms of their effectiveness in a real-world building. This methodology revealed a variety of optimization approaches and recommendations for designing dense mixed-reality auditory environments aimed at supporting cognitive map formation by visually impaired persons.

Nonvisually Accessible Video Calling (NAVC) is a prototype that detects visual conversation cues in a video call and uses audio cues to convey them to a user who is blind or low-vision. NAVC uses audio cues inspired by movie soundtracks to convey Attention, Agreement, Disagreement, Happiness, Thinking, and Surprise. When designing NAVC, we partnered with people who are blind or low-vision through a user-centered design process that included need-finding interviews and design reviews. To evaluate NAVC, we conducted a user study with 16 participants. The study provided feedback on the NAVC prototype and showed that the participants could easily discern some cues, like Attention and Agreement, but had trouble distinguishing others. The accuracy of the prototype in detecting conversation cues emerged as a key concern, especially in avoiding false positives and in detecting negative emotions, which tend to be masked in social conversations. This research identified challenges and design opportunities in using AI models to enable accessible video calling.

The PhET project is a collection of over 130 interactive simulations (or “sims”) designed to teach physics concepts to students from elementary to university levels. The sims rely heavily on visual representation, making them inaccessible to students with disabilities, including those with visual impairments. We present the theory, methods, and process behind our audio design and provide example mapping strategies from two of the simulations. We compare physical, abstract, and musical mapping strategies, noting the strengths of each. We conclude with design recommendations that have arisen in our work, and for which we think would benefit the field at large.

We present a multimodal science simulation, including visual and auditory (descriptions, sound effects, and sonifications) display. The design of each modality is described, as well as evaluation with learners with and without visual impairments. We conclude with challenges and opportunities at the intersection of multiple modalities.

Auditory user interfaces (AUIs) have been developed to support data exploration, increase engagement with arts and entertainment, and provide an alternative to visual interfaces. Standard measures of usability such as the SUS [4] and UMUX [8] can help with comparing baseline usability and user experience (UX), but the overly general nature of the questions can be confusing to users and can present problems in interpretation of the measures when evaluating an AUI. We present an efficient and effective alternative: an 11-item Auditory Interface UX Scale (BUZZ), designed to evaluate interpretation, meaning, and enjoyment of an AUI.

Auditory displays have been used extensively to enhance visual menus across diverse settings for various reasons. While standard auditory displays can be effective and help users across these settings, standard auditory displays often consist of text to speech cues, which can be time intensive to use. Advanced auditory cues including spindex and spearcon cues have been developed to help address this slow feedback issue. While these cues are most often used in English, they have also been applied to other languages, but research on using them in tonal languages, which may affect the ability to use them, is lacking. The current research investigated the use of spindex and spearcon cues in Mandarin, to determine their effectiveness in a tonal language. The results suggest that the cues can be effectively applied and used in a tonal language by untrained novices. This opens the door to future use of the cues in languages that reach a large portion of the world’s population.