No file available
To read the file of this research,
you can request a copy directly from the authors.
SoK: Data Privacy in Virtual Reality
Preprints and early-stage research may not have been peer reviewed yet.
Abstract
The adoption of virtual reality (VR) technologies has rapidly gained momentum in recent years as companies around the world begin to position the so-called "metaverse" as the next major medium for accessing and interacting with the internet. While consumers have become accustomed to a degree of data harvesting on the web, the real-time nature of data sharing in the metaverse indicates that privacy concerns are likely to be even more prevalent in the new "Web 3.0." Research into VR privacy has demonstrated that a plethora of sensitive personal information is observable by various would-be adversaries from just a few minutes of telemetry data. On the other hand, we have yet to see VR parallels for many privacy-preserving tools aimed at mitigating threats on conventional platforms. This paper aims to systematize knowledge on the landscape of VR privacy threats and countermeasures by proposing a comprehensive taxonomy of data attributes, protections, and adversaries based on the study of 68 collected publications. We complement our qualitative discussion with a statistical analysis of the risk associated with various data sources inherent to VR in consideration of the known attacks and defenses. By focusing on highlighting the clear outstanding opportunities, we hope to motivate and guide further research into this increasingly important field.
... Socio-cultural impact: Fragmentation can lead to greater control over information by governments or large corporations, limiting freedom of speech and access to information (Wang, Su and Yan, 2023;Garrido, Nair and Song, 2023). ...
The paper addresses the issue of the Metaverse's territoriality and its connection with national and international law. The study provides a brief overview of hypotheses related to the territoriality of the Metaverse and its connection with national and international law. It explores the concept of electronic jurisdiction for the Metaverse amidst the general absence of a unified transnational legal system for virtual environments. The Internet and the Metaverse are increasingly subject to the reality of fragmenting into separate segments, which can have serious consequences for global security and the economy. The risks associated with the trend of "Metaverse fragmentation" or "Splinter net"-the division of the single global internet space into isolated segments governed by different rules and technical standards-are analyzed. Innovatively, a theoretical model of a typical Metaverse is presented, potentially creating a cross-border "sandbox" for modeling technological processes, social relations, business, and legal regulation of virtual technologies to develop proposals for unifying the fundamental components of the Metaverse and simplifying cross-border interactions. The proposed Transborder Standard Model of the Metaverse is an abstract representation of systems used to understand, predict, and explain the behavior of a complex of systems known under the generalized name Metaverse. This model is characterized by a specific structure composed of modules or ecosystems that functionally differ in purpose and structure and are not connected by similar features. However, their combined application ensures the functionality of virtual environments, and their legal regulation, and can serve as the basis for electronic jurisdiction.
... Moore et al. show that encoding user tracking data in VR can be obfuscated by encoding positional data as velocity data to preserve the user's privacy in VR [42]. Also, Garrido et al. provide a SoK on data privacy in VR, systematizing knowledge on the landscape of VR privacy threats and countermeasures by proposing a taxonomy of data attributes, protections, and adversaries [18]. Notably, Kinetic signatures also can be used to infer personal attributes of VR users [46]. ...
Behavioral Biometrics in Virtual Reality (VR) enable implicit user identification by leveraging the motion data of users’ heads and hands from their interactions in VR. This spatiotemporal data forms a Kinetic Signature, which is a user-dependent behavioral biometric trait. Although kinetic signatures have been widely used in recent research, the factors contributing to their degree of identifiability remain mostly unexplored. Drawing from existing literature, this work systematically examines the influence of static and dynamic components in human motion. We conducted a user study (N = 24) with two sessions to reidentify users across different VR sports and exercises after one week. We found that the identifiability of a kinetic signature depends on its inherent static and dynamic factors, with the best combination allowing for 90.91% identification accuracy after one week had passed. Therefore, this work lays a foundation for designing and refining movement-based identification protocols in immersive environments
The rapid development of the metaverse has brought about numerous security challenges. Virtual Reality (VR) , as one of the core technologies, plays a crucial role in the metaverse. The security of VR devices directly impacts user authentication and privacy. Currently, no attention has been paid to the vulnerabilities and security risks of VR devices. This paper employs a bi-layer BiLSTM neural network to conduct a root cause analysis for user authentication and scene interaction when users enter metaverse environment using VR devices. By establishing the mapping between vulnerable VR firmware file attributes and metaverse interaction scenarios, we implement a vulnerability discovery and verification prototype called VRVul-Discovery, based on the concept of vulnerability discovery. Experiment results demonstrate that VRVul-Discovery provides high-accuracy determinations of firmware vulnerability attributes and scenarios susceptible to hijacking. In the end, the prototype system discovers seven unknown vulnerabilities, all of which are authenticated.
A brain–computer interface that decodes continuous language from non-invasive recordings would have many scientific and practical applications. Currently, however, non-invasive language decoders can only identify stimuli from among a small set of words or phrases. Here we introduce a non-invasive decoder that reconstructs continuous language from cortical semantic representations recorded using functional magnetic resonance imaging (fMRI). Given novel brain recordings, this decoder generates intelligible word sequences that recover the meaning of perceived speech, imagined speech and even silent videos, demonstrating that a single decoder can be applied to a range of tasks. We tested the decoder across cortex and found that continuous language can be separately decoded from multiple regions. As brain–computer interfaces should respect mental privacy, we tested whether successful decoding requires subject cooperation and found that subject cooperation is required both to train and to apply the decoder. Our findings demonstrate the viability of non-invasive language brain–computer interfaces.
Although Virtual Reality (VR) is certainly not a new technology, its recent adoption across several sectors beyond entertainment has led the information security research community to take note of the new cyber threats that come with it. The variety of system components presents an extensive attack surface that can be exploited. At the same time, VR’s emphasis on immersion, interaction and presence means that the user can be targeted directly, yet the use of head-mounted displays may prevent them from observing a cyber attack’s impact in their immediate physical environment. This paper presents the first taxonomic representation of VR security challenges. By systemically classifying existing VR cyber threats against existing defences in a single comparative matrix, we aim to help researchers from different backgrounds to identify key focus areas where further research would be most beneficial.
The allure of the metaverse along with Virtual Reality (VR) technologies and speed at which they are deployed may shift focus away from security and privacy fundamentals. In this work we employ classic exploitation techniques against cutting edge devices to obtain equally novel results. The unique features of the Virtual Reality landscape set the stage for our primary account of a new attack, the Man-in-the-Room (MitR). This attack, realized from a vulnerable social networking application led to both worming and botnet capabilities being adapted for VR with potential critical impacts affecting millions of users. Our work improves the state-of-the-art in Virtual Reality (VR) security and socio-technical research in VR. It shares several analytical and attacking tools, example exploits, evaluation dataset, and vulnerability signatures with the scientific and professional communities to ensure secure VR software development. The presented results demonstrate the detection and prevention of VR vulnerabilities, and raise questions in the law and policy domains pertaining to VR security and privacy.
The core Immersive Media (IM) technologies of Virtual Reality (VR) and Augmented Reality (AR) have steadily advanced over the last thirty years, enabling high fidelity experiences at consumer prices. Over the same period, networking speeds have increased dramatically, culminating in the deployment of 5G networks. Combined, these advancements greatly increase the prospects for widespread adoption of virtual and augmented worlds. Recently branded "the metaverse" by Meta and other large platforms, major corporations are currently investing billions to deploy immersive worlds that target mainstream activities from socializing and shopping to education and business. With the prospect that corporate-controlled metaverse environments proliferate society over the next decade, it is important to consider the risks to consumers and plan for meaningful regulation. This is especially true in light of the unexpected negative impact that social media platforms have had on society in recent years. The dangers of the metaverse are outlined herein along with proposals for sensible regulation.
Keywords: Metaverse, Virtual Reality, Augmented Reality, Mixed Reality, Regulation, Virtual Product Placements (VPPs), Virtual People, Virtual Spokespeople
The metaverse has the potential to extend the physical world using augmented and virtual reality technologies allowing users to seamlessly interact within real and simulated environments using avatars and holograms. Virtual environments and immersive games (such as, Second Life, Fortnite, Roblox and VRChat) have been described as antecedents of the metaverse and offer some insight to the potential socio-economic impact of a fully functional persistent cross platform metaverse. Separating the hype and “meta…” rebranding from current reality is difficult, as “big tech” paints a picture of the transformative nature of the metaverse and how it will positively impact people in their work, leisure, and social interaction. The potential impact on the way we conduct business, interact with brands and others, and develop shared experiences is likely to be transformational as the distinct lines between physical and digital are likely to be somewhat blurred from current perceptions. However, although the technology and infrastructure does not yet exist to allow the development of new immersive virtual worlds at scale - one that our avatars could transcend across platforms, researchers are increasingly examining the transformative impact of the metaverse. Impacted sectors include marketing, education, healthcare as well as societal effects relating to social interaction factors from widespread adoption, and issues relating to trust, privacy, bias, disinformation, application of law as well as psychological aspects linked to addiction and impact on vulnerable people. This study examines these topics in detail by combining the informed narrative and multi-perspective approach from experts with varied disciplinary backgrounds on many aspects of the metaverse and its transformational impact. The paper concludes by proposing a future research agenda that is valuable for researchers, professionals and policy makers alike.
Emotion recognition in smart eyewear devices is valuable but challenging. One key limitation of previous works is that the expression-related information like facial or eye images is considered as the only evidence of emotion. However, emotional status is not isolated; it is tightly associated with people's visual perceptions, especially those with emotional implications. However, little work has examined such associations to better illustrate the causes of emotions. In this paper, we study the emotionship analysis problem in eyewear systems, an ambitious task that requires classifying the user's emotions and semantically understanding their potential causes. To this end, we describe EMOShip, a deep-learning-based eyewear system that can automatically detect the wearer's emotional status and simultaneously analyze its associations with semantic-level visual perception. Experimental studies with 20 participants demonstrate that, thanks to its awareness of emotionship, EMOShip achieves superior emotion recognition accuracy compared to existing methods (80.2% vs. 69.4%) and provides a valuable understanding of the causes of emotions. Further pilot studies with 20 additional participants further motivate the potential use of EMOShip to empower emotion-aware applications, such as emotionship self-reflection and emotionship life-logging.
The metaverse promises a host of bright opportunities for business, economics, and society. Though, a number of critical aspects are still to be considered and the analysis of their impact is almost non-existent. In this paper, we provide several contributions. We start by analysing the foundations of the metaverse, later we focus on the novel privacy and security issues introduced by this new paradigm, and finally we broaden the scope of the contribution highlighting some of the far-reaching yet logical implications of the metaverse on a number of domains, not all of them in tech. Throughout the paper, we also discuss possible research directions. We believe that the provided holistic view on the foundations, technology, and issues related to the metaverse---with a focus on security and privacy---, other than being an interesting contribution on its own, could also pave the way for a few multidisciplinary research avenues.
AR/VR devices create novel issues for user privacy due to the scope, scale, and sensitivity of the information they collect. To mitigate harms, policymakers should reform the current patchwork regulatory landscape for data privacy, which fails to address some risks while over-regulating in response to others.
New generation head-mounted displays, such as VR and AR glasses, are coming into the market with already integrated eye tracking and are expected to enable novel ways of human-computer interaction in numerous applications. However, since eye movement properties contain biometric information, privacy concerns have to be handled properly. Privacy-preservation techniques such as differential privacy mechanisms have recently been applied to eye movement data obtained from such displays. Standard differential privacy mechanisms; however, are vulnerable due to temporal correlations between the eye movement observations. In this work, we propose a novel transform-coding based differential privacy mechanism to further adapt it to the statistics of eye movement feature data and compare various low-complexity methods. We extend the Fourier perturbation algorithm, which is a differential privacy mechanism, and correct a scaling mistake in its proof. Furthermore, we illustrate significant reductions in sample correlations in addition to query sensitivities, which provide the best utility-privacy trade-off in the eye tracking literature. Our results provide significantly high privacy without any essential loss in classification accuracies while hiding personal identifiers.
Virtual Reality (VR) is becoming increasingly popular both in the entertainment and professional domains. Behavioral biometrics have recently been investigated as a means to continuously and implicitly identify users in VR. VR applications can specifically benefit from this, for example, to adapt the environment and user interface as well as to authenticate users.
In this work, we conduct a lab study (N=16) to explore how accurately users can be identified during two task-driven scenarios based on their spatial movement.
We show that an identification accuracy of up to 90% is possible across sessions recorded on different days.
Moreover, we investigate the role of users' physiology on behavioral biometrics.
In particular, we virtually alter and normalize users' body proportions to examine the influence on behavior.
We find that body normalization in general increases the identification rate, in some cases by up to 38%, hence it improves the performance of identification systems.
Virtual reality (VR) is a technology that is gaining traction in the consumer market. With it comes an unprecedented ability to track body motions. These body motions are diagnostic of personal identity, medical conditions, and mental states. Previous work has focused on the identifiability of body motions in idealized situations in which some action is chosen by the study designer. In contrast, our work tests the identifiability of users under typical VR viewing circumstances, with no specially designed identifying task. Out of a pool of 511 participants, the system identifies 95% of users correctly when trained on less than 5 min of tracking data per person. We argue these results show nonverbal data should be understood by the public and by researchers as personally identifying data.
Stress can affect a person's performance and health positively and negatively. A lot of the relaxation methods have been suggested to reduce the amount of stress. This study used virtual reality (VR) video games to alleviate stress. Physiological signals captured from Electrocardiogram (ECG), galvanic skin response (GSR), and respiration (RESP) were used to determine if the subject was stressed or relaxed. Time and frequency domain features were then extracted to evaluate stress levels. Frequency domain methods such as low-frequency (LF), high-frequency (HF), LF-HF ratio (LF/HF) are considered the most effective for HRV analysis, Poincare plots are moré discerning visually and shares a 81% correlation with LF/HF ratio. GSR is associated with EDA activity, which only increases due to stress. Stress and relax were classified using Linear Discriminant Analysis (LDA), Decision Tree, Support Vector machine (SVM), Gradient Boost (GB), and Naive Bayes. GB performed the best with an accuracy of 85% after 5 fold cross validation with 100 iterations, which is admirable from a small dataset with 50 samples.
Virtual reality (VR) has advanced rapidly and is used for many entertainment and business purposes. The need for secure, transparent and non-intrusive identification mechanisms is important to facilitate users’ safe participation and secure experience. People are kinesiologically unique, having individual behavioral and movement characteristics, which can be leveraged and used in security sensitive VR applications to compensate for users’ inability to detect potential observational attackers in the physical world. Additionally, such method of identification using a user’s kinesiological data is valuable in common scenarios where multiple users simultaneously participate in a VR environment. In this paper, we present a user study (n = 15) where our participants performed a series of controlled tasks that require physical movements (such as grabbing, rotating and dropping) that could be decomposed into unique kinesiological patterns while we monitored and captured their hand, head and eye gaze data within the VR environment. We present an analysis of the data and show that these data can be used as a biometric discriminant of high confidence using machine learning classification methods such as kNN or SVM, thereby adding a layer of security in terms of identification or dynamically adapting the VR environment to the users’ preferences. We also performed a whitebox penetration testing with 12 attackers, some of whom were physically similar to the participants. We could obtain an average identification confidence value of 0.98 from the actual participants’ test data after the initial study and also a trained model classification accuracy of 98.6%. Penetration testing indicated all attackers resulted in confidence values of less than 50% (<50%), although physically similar attackers had higher confidence values. These findings can help the design and development of secure VR systems.
Technologies to measure gaze direction and pupil reactivity have become efficient, cheap, and compact and are finding increasing use in many fields, including gaming, marketing, driver safety, military, and healthcare. Besides offering numerous useful applications, the rapidly expanding technology raises serious privacy concerns. Through the lens of advanced data analytics, gaze patterns can reveal much more information than a user wishes and expects to give away. Drawing from a broad range of scientific disciplines, this paper provides a structured overview of personal data that can be inferred from recorded eye activities. Our analysis of the literature shows that eye tracking data may implicitly contain information about a user's biometric identity, gender, age, ethnicity, body weight, personality traits, drug consumption habits, emotional state, skills and abilities, fears, interests, and sexual preferences. Certain eye tracking measures may even reveal specific cognitive processes and can be used to diagnose various physical and mental health conditions. By portraying the richness and sensitivity of gaze data, this paper provides an important basis for consumer education, privacy impact assessments, and further research into the societal implications of eye tracking.
The gaze behavior of virtual avatars is critical to social presence and perceived eye contact during social interactions in Virtual Reality. Virtual Reality headsets are being designed with integrated eye tracking to enable compelling virtual social interactions. This paper shows that the near infra-red cameras used in eye tracking capture eye images that contain iris patterns of the user. Because iris patterns are a gold standard biometric, the current technology places the user's biometric identity at risk. Our first contribution is an optical defocus based hardware solution to remove the iris biometric from the stream of eye tracking images. We characterize the performance of this solution with different internal parameters. Our second contribution is a psychophysical experiment with a same-different task that investigates the sensitivity of users to a virtual avatar's eye movements when this solution is applied. By deriving detection threshold values, our findings provide a range of defocus parameters where the change in eye movements would go unnoticed in a conversational setting. Our third contribution is a perceptual study to determine the impact of defocus parameters on the perceived eye contact, attentiveness, naturalness, and truthfulness of the avatar. Thus, if a user wishes to protect their iris biometric, our approach provides a solution that balances biometric protection while preventing their conversation partner from perceiving a difference in the user's virtual avatar. This work is the first to develop secure eye tracking configurations for VR/AR/XR applications and motivates future work in the area.
For the past 20 years, researchers have investigated the use of eye tracking in security applications. We present a holistic view on gaze-based security applications. In particular, we canvassed the literature and classify the utility of gaze in security applications into a) authentication, b) privacy protection, and c) gaze monitoring during security critical tasks. This allows us to chart several research directions, most importantly 1) conducting field studies of implicit and explicit gaze-based authentication due to recent advances in eye tracking, 2) research on gaze-based privacy protection and gaze monitoring in security critical tasks which are under-investigated yet very promising areas, and 3) understanding the privacy implications of pervasive eye tracking. We discuss the most promising opportunities and most pressing challenges of eye tracking for security that will shape research in gaze-based security applications for the next decade.
Speech recordings are a rich source of personal, sensitive data that can be used to support a plethora of diverse applications,from health profiling to
biometric recognition. It is therefore essential that speech recordings are adequately protected so that they cannot be misused. Such protection, in the
form of privacy-preserving technologies, is required to ensure that: (i) the biometric profiles of a given individual
(e.g., across different biometric service operators) are unlinkable; (ii) leaked, encrypted biometric information is irreversible, and that
(iii) biometric references are renewable. Whereas many privacy-preserving technologies have been developed for other biometric characteristics,
very few solutions have been proposed to protect privacy in the case of speech signals. Despite privacy preservation this is now being mandated by
recent European and international data protection regulations. With the aim of fostering progress and collaboration between researchers in the speech,
biometrics and applied cryptography communities, this survey article provides an introduction to the field, starting with a legal perspective on privacy
preservation in the case of speech data. It then establishes the requirements for effective privacy preservation, reviews generic cryptography-based solutions,
followed by specific techniques that are applicable to speaker characterisation (biometric applications) and speech characterisation
(non-biometric applications). Glancing at non-biometrics, methods are presented to avoid function creep, preventing the exploitation of biometric information, e.g.,
to single out an identity in speech-assisted health care via speaker characterisation. In promoting harmonised research, the article also outlines common,
empirical evaluation metrics for the assessment of privacy-preserving technologies for speech data.
Metaverse, as an evolving paradigm of the next-generation Internet, aims to build a fully immersive, hyper spatiotemporal, and self-sustaining virtual shared space for humans to play, work, and socialize. Driven by recent advances in emerging technologies such as extended reality, artificial intelligence, and blockchain, metaverse is stepping from science fiction to an upcoming reality. However, severe privacy invasions and security breaches (inherited from underlying technologies or emerged in the new digital ecology) of metaverse can impede its wide deployment. At the same time, a series of fundamental challenges (e.g., scalability and interoperability) can arise in metaverse security provisioning owing to the intrinsic characteristics of metaverse, such as immersive realism, hyper spatiotemporality, sustainability, and heterogeneity. In this paper, we present a comprehensive survey of the fundamentals, security, and privacy of metaverse. Specifically, we first investigate a novel distributed metaverse architecture and its key characteristics with ternary-world interactions. Then, we discuss the security and privacy threats, present the critical challenges of metaverse systems, and review the state-of-the-art countermeasures. Finally, we draw open research directions for building future metaverse systems.
Augmented (AR) and Virtual Reality (VR) have emerged across several industries and are steadily progressing to a comprehensive level of user adoption. However, while different companies are introducing AR/VR devices and applications, little or no attention is being placed on evaluating their security. To this aid, we collected the most commonly available 10 AR/VR devices and applications, including Hololens, Oculus, Google Glass, Valve Index, HTC Vive, Raptor AR, Psious (Amelia), Magic Leap, Epson Moverio, and IKEA Place AR. We then performed a security and privacy evaluation of the devices and their associated platforms while focusing on-device authentication, user profiling, access control, database security, and others. Our findings inform the limitations of these identified devices and applications and suggest areas where enhancements can be provided to ensure users' better, privacy-preserving, more secure experience.
IoT data markets in public and private institutions have become increasingly relevant in recent years because of their potential to improve data availability and unlock new business models. However, exchanging data in markets bears considerable challenges related to disclosing sensitive information. Despite considerable research focused on different aspects of privacy-enhancing data markets for the IoT, none of the solutions proposed so far seems to find a practical adoption. Thus, this study aims to organize the state-of-the-art solutions, analyze and scope the technologies that have been suggested in this context, and structure the remaining challenges to determine areas where future research is required. To accomplish this goal, we conducted a systematic literature review on privacy enhancement in data markets for the IoT, covering 50 publications dated up to July 2020, and provided updates with 24 publications dated up to May 2022. Our results indicate that most research in this area has emerged only recently, and no IoT data market architecture has established itself as canonical. Existing solutions frequently lack the required combination of anonymization and secure computation technologies. Furthermore, there is no consensus on the appropriate use of blockchain technology for IoT data markets and a low degree of leveraging existing libraries or reusing generic data market architectures. We also identified significant challenges remaining, such as the copy problem and the recursive enforcement problem that – while solutions have been suggested to some extent – are often not sufficiently addressed in proposed designs. We conclude that privacy-enhancing technologies need further improvements to positively impact data markets so that, ultimately, the value of data is preserved through data scarcity and users’ privacy and businesses-critical information are protected.
This article explores the influence of personality on physiological data while driving in reaction to near crashes and risky situations using Machine Learning (ML). The objective is to improve the driving assistance systems in considering drivers’ characteristics.
Methods
Physiological and behavioral data were recorded in sixty-three healthy volunteers during risky urban situations and analyzed using 5 ML algorithms to discriminate the driver’s personality according to Big Five Inventory and STAI trait. Seven step process was performed including data pre-processing, Electrodermal Activity (EDA) time windows selection (one by one backward and forward approach comparison with a pseudo-wrapped), personality traits assessment, input algorithms parameters optimization, algorithm comparison and personality trait cluster prediction. ROC Area Under the Curve (AUC) was used to describe improvement.
Results/discussion
The pseudo-wrapped/all possibilities method comparison resulted in 8.3% on average for all personality traits and all algorithms (% of ROC AUC of backward and forward approach). The ROC AUC for the detection of the personality ranged between 0.968 to 0.974 with better detection of Openness, Agreeability and Neuroticism. Use of association between Neuroticism, Extraversion and Conscientiousness previously defined in the literature slightly improve personality detection (maximum ROC AUC of 0.961 to 0.993 for cluster). Results are discussed in terms of contribution to driving aids.
Conclusion
This study is one of the first to use machine learning techniques to detect personality traits using behavioral and physiological measures in a driving context. Additionally, it questions input parameters optimization approach, time windows selection, as well as clustering and association of personality trait for detection improvement.
Mobile augmented reality systems are becoming increasingly common and powerful, with applications in such domains as healthcare, manufacturing, education, and more. This rise in popularity is thanks in part to the functionalities offered by commercially available vision libraries such as ARCore, Vuforia, and Google’s ML Kit; however, these libraries also give rise to the possibility of a hidden operations threat , that is, the ability of a malicious or incompetent application developer to conduct additional vision operations behind the scenes of an otherwise honest AR application without alerting the end user. In this paper, we present the privacy risks associated with the hidden operations threat, and propose a framework for application development and runtime permissions targeted specifically at preventing the execution of hidden operations. We follow this with a set of experimental results, exploring the feasibility and utility of our system in differentiating between user-expectation-compliant and non-compliant AR applications during runtime testing, for which preliminary results demonstrate accuracy of up to 71%. We conclude with a discussion of open problems in the areas of software testing and privacy standards in mobile AR systems.
Current Social VR literature provides limited insight on one of the most critical behaviors for developing and maintaining interpersonal relationships: self-disclosure. Therefore, we present an online survey (N = 126) investigating how users disclose personal information to each other in Social VR. Our results indicate that many participants see in Social VR access to authentic connections with others despite tending towards skepticism and privacy concerns. Most users disclose sexuality-related information, lifestyle preferences, and personal goals. In contrast, information that breaks anonymity, such as real names and more intimate aspects of oneself, are shared less commonly. Thereby, self-disclosure decisions depend on factors like the relationship to or age of disclosure recipients, the privacy of a virtual environment, the group size, or the activity context, and is driven by different goals, i.a., relational development or exploration of oneself. These insights advance the understanding of current Social VR users and their behavior by directing future research on self-disclosure-based relationship building in Social VR and outlying broader design implications for the future metaverse.
Previously, 3D data---particularly, spatial data---have primarily been utilized in the field of geo-spatial analyses, or robot navigation (e.g. self-automated cars) as 3D representations of geographical or terrain data (usually extracted from lidar). Now, with the increasing user adoption of augmented, mixed, and virtual reality (AR/MR/VR; we collectively refer to as MR) technology on user mobile devices, spatial data has become more ubiquitous. However, this ubiquity also opens up a new threat vector for adversaries: aside from the traditional forms of mobile media such as images and video, spatial data poses additional and, potentially, latent risks to users of AR/MR/VR. Thus, in this work, we analyse MR spatial data using various spatial complexity metrics---including a cosine similarity-based, and a Euclidean distance-based metric---as heuristic or empirical measures that can signify the inference risk a captured space has. To demonstrate the risk, we utilise 3D shape recognition and classification algorithms for spatial inference attacks over various 3D spatial data captured using mobile MR platforms: i.e. Microsoft HoloLens, and Android with Google ARCore. Our experimental evaluation and investigation shows that the cosine similarity-based metric is a good spatial complexity measure of captured 3D spatial maps and can be utilised as an indicator of spatial inference risk.
Eye-tracking technology is being increasingly integrated into mixed reality devices. Although critical applications are being enabled, there are significant possibilities for violating user privacy expectations. We show that there is an appreciable risk of unique user identification even under natural viewing conditions in virtual reality. This identification would allow an app to connect a user's personal ID with their work ID without needing their consent, for example. To mitigate such risks we propose a framework that incorporates gatekeeping via the design of the application programming interface and via software-implemented privacy mechanisms. Our results indicate that these mechanisms can reduce the rate of identification from as much as 85% to as low as 30%. The impact of introducing these mechanisms is less than 1.5° error in gaze position for gaze prediction. Gaze data streams can thus be made private while still allowing for gaze prediction, for example, during foveated rendering. Our approach is the first to support privacy-by-design in the flow of eye-tracking data within mixed reality use cases.
Privacy is a feature inherent to the use of cash. With steadily increasing market shares of digital payment platforms, privacy in payments may no longer be attainable in the future. We explore the potential welfare impacts of reductions in privacy in payments. In our framework, firms may use data collected through payments to price discriminate future consumers. A public good aspect arises because individuals do not internalize the full cost of failing to protect their privacy and reduce social welfare by suboptimally choosing not to protect their privacy in payments. We discuss potential remedies, including the issuance of electronic cash.
VR telepresence consists of interacting with another human in a virtual space represented by an avatar. Today most avatars are cartoon-like, but soon the technology will allow video-realistic ones. This paper aims in this direction, and presents Modular Codec Avatars (MCA), a method to generate hyper-realistic faces driven by the cameras in the VR headset. MCA extends traditional Codec Avatars (CA) by replacing the holistic models with a learned modular representation. It is important to note that traditional person-specific CAs are learned from few training samples, and typically lack robustness as well as limited expressiveness when transferring facial expressions. MCAs solve these issues by learning a modulated adaptive blending of different facial components as well as an exemplar-based latent alignment. We demonstrate that MCA achieves improved expressiveness and robustness w.r.t to CA in a variety of real-world datasets and practical scenarios. Finally, we showcase new applications in VR telepresence enabled by the proposed model.
Video-based eye trackers estimate gaze based on eye images/videos. As security and privacy concerns loom over technological advancements, tackling such challenges is crucial. We present a new approach to handle privacy issues in eye videos by replacing the current identifiable iris texture with a different iris template in the video capture pipeline based on the Rubber Sheet Model. We extend to image blending and median-value representations to demonstrate that videos can be manipulated without significantly degrading segmentation and pupil detection accuracy.
Virtual reality (VR) is a rapidly advancing technology with diverse applications. VR systems offer an immersive, life-like virtual experience by rendering interactive views on a head-mounted display. VR systems are vulnerable to various types of attacks, which could have dire consequences as they are designed to replace our perception of the physical world. However, there have been few studies on the security and integrity issues of the VR systems. We focus on the popular HTC Vive VR system and propose novel attack methods on blocking and manipulating its tracking subsystem. It is shown that simple attacks can jam or even manipulate the entire position and pose tracking process. Possible countermeasures are suggested to make VR systems safer and more secure.
In recent years, rapid technological advancements in smart devices and their usage in a wide range of applications exponentially increases the data generated from these devices. So, the traditional data analytics techniques may not be able to handle this extreme volume of data known as Big Data (BD) generated by different devices. However, this exponential increase of data opens the doors for the different type of attackers to launch various attacks by exploiting various vulnerabilities (SQL injection, OS fingerprinting, malicious code execution, etc.) during data analytics. Motivated from the aforementioned discussion, in this paper, we explored Machine Learning (ML) and Deep Learning (DL)-based models and techniques which are capable off to identify and mitigate both the known as well as unknown attacks. ML and DL-based techniques have the capabilities to learn from the traffic pattern using training and testing datasets in the extensive network domains to make intelligent decisions concerning attack identification and mitigation. We also proposed a DL and ML-based Secure Data Analytics (SDA) architecture to classify normal or attack input data. A detailed taxonomy of SDA is abstracted into a threat model. This threat model addresses various research challenges in SDA using multiple parameters such as-efficiency, latency, accuracy, reliability, and attacks launched by the attackers. Finally, a comparison of existing SDA proposals with respect to various parameters is presented, which allows the end users to select one of the SDA proposals in comparison to its merits over the others.
Virtual reality exposure therapy (VRET) can have a significant impact towards assessing and potentially treating various anxiety disorders. One of the main strengths of VRET systems is that they provide an opportunity for a psychologist to interact with virtual 3D environments and change therapy scenarios according to the individual patient's needs. However, to do this efficiently the patient's anxiety level should be tracked throughout the VRET session. Therefore, in order to fully use all advantages provided by the VRET system, a mental stress detection system is needed. The patient's physiological signals can be collected with wearable biofeedback sensors. Signals like blood volume pressure (BVP), galvanic skin response (GSR), and skin temperature can be processed and used to train the anxiety level classification models. In this paper, we combine VRET with mental stress detection and highlight potential uses of this kind of VRET system. We discuss and present a framework for anxiety level recognition, which is a part of our developed cloud-based VRET system. Physiological signals of 30 participants were collected during VRET-based public speaking anxiety treatment sessions. The acquired data were used to train a four-level anxiety recognition model (where each level of 'low', 'mild', 'moderate', and 'high' refer to the levels of anxiety rather than to separate classes of the anxiety disorder). We achieved an 80.1% cross-subject accuracy (using leave-one-subject-out cross-validation) and 86.3% accuracy (using 10 × 10 fold cross-validation) with the signal fusion-based support vector machine (SVM) classifier.
We have seen a rise in mixed (MR) and augmented reality (AR) applications and devices in recent years. Subsequently, we have become familiar with the sensing power of these applications and devices, and we are only starting to realize the nascent risks that these technology puts over our privacy and security. Current privacy protection measures are primarily aimed towards known and well-utilised data types (i.e. location, on-line activity, biometric, and so on) while a few works have focused on looking into the security and privacy risks of and providing protection on MR data, particularly on 3D MR data. In this work, we primarily reveal the privacy leakage from released 3D MR data and how the leakage persist even after implementing spatial generalizations and abstractions. Firstly, we formalize the spatial privacy problem in 3D mixed reality data as well as the adversary model. Then, we demonstrate through an inference model how adversaries can identify 3D spaces and, potentially, infer more spatial information. Moreover, we also demonstrate how compact 3D MR Data can be in terms of memory usage which allows adversaries to create lightweight 3D inference models of user spaces.
Physical keyboards are common peripherals for personal computers and are efficient standard text entry devices. Recent research has investigated how physical keyboards can be used in immersive head-mounted display-based Virtual Reality (VR). So far, the physical layout of keyboards has typically been transplanted into VR for replicating typing experiences in a standard desktop environment. In this paper, we explore how to fully leverage the immersiveness of VR to change the input and output characteristics of physical keyboard interaction within a VR environment. This allows individual physical keys to be reconfigured to the same or different actions and visual output to be distributed in various ways across the VR representation of the keyboard. We explore a set of input and output mappings for reconfiguring the virtual presentation of physical keyboards and probe the resulting design space by specifically designing, implementing and evaluating nine VR-relevant applications: emojis, languages and special characters, application shortcuts, virtual text processing macros, a window manager, a photo browser, a whack-a-mole game, secure password entry and a virtual touch bar. We investigate the feasibility of the applications in a user study with 20 participants and find that, among other things, they are usable in VR. We discuss the limitations and possibilities of remapping the input and output characteristics of physical keyboards in VR based on empirical findings and analysis and suggest future research directions in this area.