Dieter Schmalstieg

Dieter Schmalstieg
Graz University of Technology | TU Graz · Institute of Computer Graphics and Knowledge Visualisation

PhD in computer science

About

562
Publications
232,704
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
15,334
Citations

Publications

Publications (562)
Preprint
Full-text available
The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality display, is the main player in the recent boost in medical augmented reality research. In medical settings, the HoloLens enables the physician to obtain immediate insight into patient information, directly overlaid with their view of the clinical scena...
Article
Among the most compelling applications of Augmented Reality are spatially registered tutorials. The effort of creating such instructions remains one of the obstacles precluding a wider use. We propose a system that is capable of extracting 3D instructions in a completely model-free manner from demonstrations, based on volumetric changes. The instru...
Article
We present a registration method relying on geometric constraints extracted from parametric primitives contained in 3D parametric models. Our method solves the registration in closed-form from three line-to-line, line-to-plane or plane-to-plane correspondences. The approach either works with semantically segmented RGB-D scans of the scene or with t...
Article
Diminished Reality (DR) propagates pixels from a keyframe to subsequent frames for real-time inpainting. Keyframe selection has a significant impact on the inpainting quality, but untrained users struggle to identify good keyframes. Automatic selection is not straightforward either, since no previous work has formalized or verified what determines...
Article
Commonly used image‐space layouts of shading points, such as used in deferred shading, are strictly view‐dependent, which restricts efficient caching and temporal amortization. In contrast, texture‐space layouts can represent shading on all surface points and can be tailored to the needs of a particular application. However, the best grouping of sh...
Article
We present RagRug, an open-source toolkit for situated analytics. The abilities of RagRug go beyond previous immersive analytics toolkits by focusing on specific requirements emerging when using augmented reality (AR) rather than virtual reality. RagRug combines state of the art visual encoding capabilities with a comprehensive physical-virtual mod...
Conference Paper
We introduce a system capable of generating interactive Augmented Reality guitar tutorials by parsing common digital guitar tablature and by capturing the performance of an expert using a multi-camera array. Instructions are presented to the user in an Augmented Reality application using either an abstract visualization, a 3D virtual hand, or a 3D...
Article
This work introduces the first approach to video see-through mixed reality with full support for focus cues. By combining the flexibility to adjust the focus distance found in varifocal designs with the robustness to eye-tracking error found in multifocal designs, our novel display architecture reliably delivers focus cues over a large workspace. I...
Preprint
Full-text available
Traditional convolutional neural network (CNN) methods rely on dense tensors, which makes them suboptimal for spatially sparse data. In this paper, we propose a CNN model based on sparse tensors for efficient processing of large and sparse medical images. In contrast to a dense CNN that takes the entire voxel grid as input, a sparse CNN processes o...
Preprint
Traditional convolutional neural network (CNN) methods rely on dense tensors, which makes them suboptimal for spatially sparse data. In this paper, we propose a CNN model based on sparse tensors for efficient processing of large and sparse medical images. In contrast to a dense CNN that takes the entire voxel grid as input, a sparse CNN processes o...
Preprint
Traditional convolutional neural network (CNN) methods rely on dense tensors, which makes them suboptimal for spatially sparse data. In this paper, we propose a CNN model based on sparse tensors for efficient processing of large and sparse medical images. In contrast to a dense CNN that takes the entire voxel grid as input, a sparse CNN processes o...
Chapter
In this paper we present an improved version of MAUI [9] (MAUI - Maintenance Assistance User Interface), extending the user-study, giving detailed insight into the implementations and introducing a new User-Interface for mobile use. MAUI is a novel take on tele-assisted tasks on cyber-physical systems. In its core we do not only provide real-time c...
Conference Paper
Surgical navigation requires tracking of instruments with respect to the patient. Conventionally, tracking is done with stationary cameras, and the navigation information is displayed on a stationary display. In contrast, an augmented reality (AR) headset can superimpose surgical navigation information directly in the surgeon’s view. However, AR ne...
Preprint
Full-text available
Objective: Surveillance imaging of chronic aortic diseases, such as dissections, relies on obtaining and comparing cross-sectional diameter measurements at predefined aortic landmarks, over time. Due to a lack of robust tools, the orientation of the cross-sectional planes is defined manually by highly trained operators. We show how manual annotatio...
Article
Civil engineering is a primary domain for new augmented reality technologies. In this work, the area of subsurface utility engineering is revisited, and new methods tackling well-known, yet unsolved problems are presented. We describe our solution to the outdoor localization problem, which is deemed one of the most critical issues in outdoor augmen...
Article
A fast and fully automatic design of 3D printed patient-specific cranial implants is highly desired in cranioplasty - the process to restore a defect on the skull. We formulate skull defect restoration as a 3D volumetric shape completion task, where a partial skull volume is completed automatically. The difference between the completed skull and th...
Conference Paper
Full-text available
Immersive Analytics is a quickly evolving field that unites several areas such as visualisation, immersive environments, and human-computer interaction to support human data analysis with emerging technologies. This research has thrived over the past years with multiple workshops, seminars, and a growing body of publications, spanning several confe...
Article
The aim of this paper is to provide a comprehensive overview of the MICCAI 2020 AutoImplant Challenge1. The approaches and publications submitted and accepted within the challenge will be summarized and reported, highlighting common algorithmic trends and algorithmic diversity. Furthermore, the evaluation results will be presented, compared and dis...
Article
Full-text available
This paper introduces a design framework for mixed reality urban exploration (MRUE), based on a concrete implementation in a historical city. The framework integrates different modalities, such as virtual reality (VR), augmented reality (AR), and haptics-audio interfaces, as well as advanced features such as personalized recommendations, social exp...
Article
Temporal coherence has the potential to enable a huge reduction of shading costs in rendering. Existing techniques focus either only on spatial shading reuse or cannot adaptively choose temporal shading frequencies. We find that temporal shading reuse is possible for extended periods of time for a majority of samples, and we show under which circum...
Article
Many professionals, like journalists, writers, or consultants, need to acquire information from various sources, make sense of this unstructured evidence, structure their observations, and finally create and deliver their product, such as a report or a presentation. In formative interviews, we found that tools allowing structuring of observations a...
Article
Background and Objective: Augmented reality (AR) can help to overcome current limitations in computer assisted head and neck surgery by granting “X-ray vision” to physicians. Still, the acceptance of AR in clinical applications is limited by technical and clinical challenges. We aim to demonstrate the benefit of a marker-free, instant calibration A...
Conference Paper
The rise of IoT-ready devices is supported through well-established web concepts for communication and analytics, but interaction yet remains in the world of web browsers and screen-based 2D interaction during times of tablet and smartphone popularity. Transforming IoT interaction concepts into 3D for future exploitation with head-worn XR devices i...
Chapter
Aortic dissection (AD) is a condition of the main artery of the human body, resulting in the formation of a new flow channel, or false lumen (FL). The disease is usually diagnosed with a computed tomography angiography (CTA) scan during the acute phase. A better understanding of the causes of AD requires knowledge of aortic geometry prior to the ev...
Technical Report
Full-text available
Aortic disease is a broad term, which includes aortic aneurysms, dissections, intramural hematoma and other conditions. The management of these conditions varies from long-term pharmacological treatment to immediate aortic surgery. Although different decisions can be made on specific cases, surgical intervention is advised for patients with an aort...
Conference Paper
Aortic dissection (AD) is a condition of the main artery of the human body, resulting in the formation of a new flow channel, or false lumen (FL). The disease is usually diagnosed with a computed tomography angiography (CTA) scan during the acute phase. A better understanding of the causes of AD requires knowledge of aortic geometry prior to the ev...
Preprint
Full-text available
We present a systematic review of 458 papers that report on evaluations in mixed and augmented reality (MR/AR) published in ISMAR, CHI, IEEE VR, and UIST over a span of 11 years (2009-2019). Our goal is to provide guidance for future evaluations of MR/AR approaches. To this end, we characterize publications by paper type (e.g., technique, design st...
Article
State-of-the-art methods for diminished reality propagate pixel information from a keyframe to subsequent frames for real-time inpainting. However, these approaches produce artifacts, if the scene geometry is not sufficiently planar. In this article, we present InpaintFusion, a new real-time method that extends inpainting to non-planar scenes by co...
Conference Paper
Full-text available
Remote assistance represents an important use case for mixed reality. With the rise of handheld and wearable devices, remote assistance has become practical in the wild. However, spontaneous provisioning of remote assistance requires an easy, fast and robust approach for capturing and sharing of unprepared environments. In this work, we make a case...
Article
Full-text available
Indoor reconstruction using depth camera algorithms (e.g., InfiniTAMv3) is becoming increasingly popular. Simple reconstruction methods solely use the frames of the depth camera, leaving any imagery from the adjunct RGB camera untouched. Recent approaches also incorporate color camera information to improve consistency. However, the results heavily...
Poster
Full-text available
Medical augmented reality (AR) offers a more intuitive mapping from 3D imaging to the patient, natural 3D interaction and increased perception of 3D structures, to physicians. Image-topatient registration is the key enabling technology for such AR systems. Related works use manual alignment of virtual content, marker-based registration or external...
Chapter
Full-text available
In the treatment of head and neck cancer, physicians can benefit from augmented reality in preparing and executing treatment. We present a system allowing a physician wearing an untethered augmented reality headset to see medical visualizations precisely overlaid onto the patient. Our main contribution is a strategy for markerless registration of 3...
Article
Full-text available
Exploration of challenging indoor environments is a demanding task. While automation with aerial robots seems a promising solution, fully autonomous systems still struggle with high-level cognitive tasks and intuitive decision making. To facilitate automation, we introduce a novel teleoperation system with an aerial telerobot that is capable of han...
Conference Paper
Full-text available
In the treatment of head and neck cancer, physicians can benefit from augmented reality in preparing and executing treatment. We present a system allowing a physician wearing an untethered augmented reality headset to see medical visualizations precisely overlaid onto the patient. Our main contribution is a strategy for markerless registration of 3...
Article
We propose an algorithm for generating an unstructured lumigraph in real-time from an image stream. This problem has important applications in mixed reality, such as telepresence, interior design or as-built documentation. Unlike conventional texture optimization in structure from motion, our method must choose views from the input stream in a stri...
Article
Full-text available
Development and verification of real-time algorithms for robotic total stations usually require hard-ware-in-the-loop approaches, which can be complex and time-consuming. Simulator-in-the-loop can be used instead, but the design of a simulation environment and sufficient detailed modeling of the hardware are required. Typically, device specificatio...
Conference Paper
Streaming high quality rendering for virtual reality applications requires minimizing perceived latency. Shading Atlas Streaming (SAS) [Mueller et al. 2018] is a novel object-space rendering framework suitable for streaming virtual reality content. SAS decouples server-side shading from client-side rendering, allowing the client to perform framerat...
Preprint
Full-text available
We present DetectFusion, an RGB-D SLAM system that runs in real-time and can robustly handle semantically known and unknown objects that can move dynamically in the scene. Our system detects, segments and assigns semantic class labels to known objects in the scene, while tracking and reconstructing them even when they move independently in front of...
Article
In this paper, we introduce the CPatch, a curved primitive that can be used to construct arbitrary vector graphics. A CPatch is a generalization of a 2D polygon: Any number of curves up to a cubic degree bound a primitive. We show that a CPatch can be rasterized efficiently in a hierarchical manner on the GPU, locally discarding irrelevant portions...
Conference Paper
Full-text available
The latest generation of consumer market Head-mounted displays (HMD) now include self-contained inside-out tracking of head motions, which makes them suitable for mobile applications. However, 3D tracking of input devices is either not included at all or requires to keep the device in sight, so that it can be observed from a sensor mounted on the H...
Chapter
Performing accurate measurements on non-planar targets using a robotic total station in reflectorless mode is prone to errors. Besides requiring a fully reflected laser beam of the electronic distance meter, a proper orientation of the pan-tilt unit is required for each individual accurate 3D point measurement. Dominant physical 3D structures like...
Conference Paper
Streaming high quality rendering for virtual reality applications requires minimizing perceived latency. We introduce Shading Atlas Streaming (SAS), a novel object-space rendering framework suitable for streaming virtual reality content. SAS decouples server-side shading from client-side rendering, allowing the client to perform framerate upsamplin...
Conference Paper
Full-text available
Having a virtual body can increase embodiment in virtual reality (VR) applications. However, comsumer-grade VR falls short of delivering sufficient sensory information for full-body motion capture. Consequently, most current VR applications do not even show arms, although they are often in the field of view. We address this shortcoming with a novel...
Article
Wwelcome to the November 2018 issue of the IEEE Transactions on Visualization and Computer Graphics (TVCG) . This issue contains selected papers accepted at the IEEE International Symposium on Mixed and Augmented Reality (ISMAR), held this year in Munich, Germany, from October 16 to October 20, 2018.
Chapter
This chapter introduces the concept of situated analytics that employs data representations organized in relation to germane objects, places, and persons for the purpose of understanding, sensemaking, and decision-making. The components of situated analytics are characterized in greater detail, including the users, tasks, data, representations, int...
Article
Due to its flexibility, compute mode is becoming more and more attractive as a way to implement many of the algorithms part of a state-of-the-art rendering pipeline. A key problem commonly encountered in graphics applications is streaming vertex and geometry processing. In a typical triangle mesh, the same vertex is on average referenced six times....
Article
In this paper, we question the premise that graphics hardware uses a post-transform cache to avoid redundant vertex shader invocations. A large body of existing work on optimizing indexed triangle sets for rendering speed is based upon this widely-accepted assumption. We conclusively show that this assumption does not hold up on modern graphics har...
Article
In this paper, we present a real-time graphics pipeline implemented entirely in software on a modern GPU. As opposed to previous work, our approach features a fully-concurrent, multi-stage, streaming design with dynamic load balancing, capable of operating efficiently within bounded memory. We address issues such as primitive order, vertex reuse, a...
Poster
Full-text available
Deep learning with neural networks is an increasingly important topic for research and economic purposes. Software giants use deep networks for the development of their latest technological gadgets. Daily examples are Facebook’s face detection, Apple’s speech recognition Siri or Google Translate, which all comprise deep learning algorithms. The mo...
Conference Paper
This contribution presents the automatic segmentation of the lower jawbone (mandible) in humans’ computed tomography (CT) images with the support of trained deep learning networks. CT acquisitions from the mandible frequently include radiological artifacts e.g. from metal dental restorations, ostheosynthesis materials or include trauma related free...
Conference Paper
Harnessing the power of massively parallel devices like the graphics processing unit (GPU) is difficult for algorithms that show dynamic or inhomogeneous workloads. To achieve high performance, such advanced algorithms require scalable, concurrent queues to collect and distribute work. We show that previous queuing approaches are unfit for this tas...
Preprint
Compute-mode rendering is becoming more and more attractive for non-standard rendering applications, due to the high flexibility of compute-mode execution. These newly designed pipelines often include streaming vertex and geometry processing stages. In typical triangle meshes, the same transformed vertex is on average required six times during rend...
Article
Full-text available
Introduction Computer assisted technologies based on algorithmic software segmentation are an increasing topic of interest in complex surgical cases. However—due to functional instability, time consuming software processes, personnel resources or licensed-based financial costs many segmentation processes are often outsourced from clinical centers t...
Article
This special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG) contains the 29 full papers selected for the IEEE Virtual Reality and 3D User Interfaces (IEEE VR 2018) Conference held in Reutlingen, Germany, March 18-22, 2017. Since its inception in 1993, IEEE VR has been the premier venue to present new research results in th...
Article
Full-text available
Three-dimensional (3D) crime scene documentation using 3D scanners and medical imaging modalities like computed tomography (CT) and magnetic resonance imaging (MRI) are increasingly applied in forensic casework. Together with digital photography, these modalities enable comprehensive and non-invasive recording of forensically relevant information r...
Conference Paper
Full-text available
Accurate segmentation and measurement of brain tumors plays an important role in clinical practice and research, as it is critical for treatment planning and monitoring of tumor growth. However, brain tumor segmentation is one of the most challenging tasks in medical image analysis. Since manual segmentations are subjective, time consuming and neit...
Poster
Full-text available
Virtual Reality (VR) is a promising future. It sets up a virtual environment providing an (audio)visual user experience by creating an artificial world where a user is directly placed into. Extensible by haptic systems, it allows to simulate lifelike experiences to any user. Over the last years, VR hardware has more and more turned into something t...
Poster
Full-text available
Accurate segmentation and measurement of brain tumors plays an important role in clinical practice and research, as it is critical for treatment planning and monitoring of tumor growth. However, brain tumor segmentation is one of the most challenging tasks in medical image analysis. Since manual segmentations are subjective, time consuming and neit...
Conference Paper
Accurate segmentation of medical images is a key step in medical image processing. As the amount of medical images obtained in diagnostics, clinical studies and treatment planning increases, automatic segmentation algorithms become increasingly more important. Therefore, we plan to develop an automatic segmentation approach for the urinary bladder...