Joaquim Armando Pires Jorge

Inesc-ID · Visualization and Intelligent Multimodal Interfaces (VIMMI)

Citation: Serras, J.; Duchowski, A.; Nobre, I.; Moreira, C.; Maciel, A.; Jorge, J. Immersive Virtual Colonography Viewer for Colon Growths Diagnosis: Design and Think-Aloud Study. Multimodal Technol. Interact. 2024, 8, 40. https:// Abstract: Desktop-based virtual colonoscopy is a proven and accurate process for identifying colon abnormalities. Howe...

Citation: Hsieh, C.; Luís, A.; Neves, J.; Nobre, I.B.; Sousa, S.C.; Ouyang, C.; Jorge, J.; Moreira, C. EyeXNet: Enhancing Abnormality Detection and Diagnosis via Eye-Tracking and X-ray Fusion. Mach. Learn. Knowl. Extr. 2024, 6, 1055-1071. https:// Abstract: Integrating eye gaze data with chest X-ray images in deep learning (DL) has led to contradic...

Dive deep into the nuanced debate of integrating generative AI in education. We explore both the transformative potential and the pitfalls of using AI tools like ChatGPT, LLama, and Gemini in academic settings. Our piece offers a critical analysis, balancing the prospects of democratized learning against the risks of cognitive decline and misinform...

Fake news has been linked to the rise of psychological disorders, the increased disbelief in science, and the erosion of democracy and freedom of speech. Online social networks are arguably the main vehicle of fake news spread. Educating online users with explanations is one way of preventing this spread. Understanding how online belief is formed a...

X-ray imaging plays a crucial role in diagnostic medicine. Yet, a significant portion of the global population lacks access to this essential technology due to a shortage of trained radiologists. Eye-tracking data and deep learning models can enhance X-ray analysis by mapping expert focus areas, guiding automated anomaly detection, optimizing workf...

Recent research has leveraged peer assessment as a grading system tool where learners are involved in learning and evaluation. However, there is limited knowledge regarding individual differences, such as personality, in peer assessment tasks. We analyze how personality factors affect the peer assessment dynamics of a semester-long remote learning...

Visual analytics combines automated analysis techniques with interactive visualizations for effective understanding, reasoning, and decision-making on complex data. However, accurately classifying sentiments and stances in sentiment analysis remains challenging due to ambiguity and individual differences. This survey examines 35 papers published be...

This study investigates the effects of including patients’ clinical information on the performance of deep learning (DL) classifiers for disease location in chest X-ray images. Although current classifiers achieve high performance using chest X-ray images alone, consultations with practicing radiologists indicate that clinical data is highly inform...

Informal caregivers play an essential role in caring for persons who require assistance and in managing the health of their loved ones. Unfortunately, they need more health, leisure, and relaxation time. Nature interaction is one of many kinds of self-care intervention. It has long been regarded as a refreshing break from stressful routines, and re...

Radiologists are trained professionals who use medical images to obtain clinically relevant information. However, little is known about visual search patterns and radiologists' strategies during medical image analysis. Thus, there is a current need for guidelines to specify optimal visual search routines commonly used by radiologists. Identifying t...

The recent pandemic, war, and oil crises have caused many to reconsider their need to travel for education, training, and meetings. Providing assistance and training remotely has thus gained importance for many applications, from industrial maintenance to surgical telemonitoring. Current solutions such as video conferencing platforms lack essential...

Highly complex and dense models of 3D objects have recently become indispensable in digital industries. Mesh decimation then plays a crucial role in the production pipeline to efficiently get visually convincing yet compact expressions of complex meshes. However, the current pipeline typically does not allow artists control the decimation process,...

This study aims to investigate the effects of including patients' clinical information on the performance of deep learning (DL) classifiers for disease location in chest X-ray images. Although current classifiers achieve high performance using chest X-ray images alone, our interviews with radiologists indicate that clinical data is highly informati...

Remote collaborative work has become pervasive in many settings, from engineering to medical professions. Users are immersed in virtual environments and communicate through life-sized avatars that enable face-to-face collaboration. Within this context, users often collaboratively view and interact with virtual 3D models, for example, to assist in d...

There is a strong demand for virtual reality (VR) to bring quality healthcare to underserved populations. This paper addresses this need with the design and prototype of SURVIVRS: Surround Video-Based Virtual Reality for Surgery Guidance. SURVIVRS allows a remote specialist to guide a local surgery team through a virtual reality (VR) telepresence i...

This paper proposes a novel multimodal DL architecture incorporating medical images and eye-tracking data for abnormality detection in chest x-rays. Our results show that applying eye gaze data directly into DL architectures does not show superior predictive performance in abnormality detection chest X-rays. These results support other works in the...

Diabetic Foot is one of the most common complications of Diabetes Mellitus and the leading non-traumatic cause of lower-limb amputations worldwide. These complications are preventable with early diagnosis and timely care. However, even in the context of expanding primary health care, this problem continues to increase, which suggests a gap in the t...

Desktop-based virtual colonoscopy has been proven to be an asset in the identification of colon anomalies. The process is accurate, although time-consuming. The use of immersive interfaces for virtual colonoscopy is incipient and not yet understood. In this work, we present a new design exploring elements of the VR paradigm to make the immersive an...

Monitoring and analyzing physical activity is becoming an important task in both clinical and non-clinical settings. To accomplish this desideratum, stick figures are often used as abstractions of human poses and movements by representing body segments as straight lines (sticks). Despite their straightforwardness, this minimalist representation is...

The main challenge in higher education is student retention. Different methods have been proposed to overcome this challenge. In this paper, we propose a method for the early prediction of student final grades for a course using only their performance data in the current semester. It assists students to analyze how much effort they need to put into...

Virtual Reality has become an important educational tool, due to the pandemic and increasing globalization of education. This paper presents a framework for teaching Virtual Anatomy at the university level. Virtual classes have become a staple of today's curricula because of the isolation and quarantine requirements and the increased international...

Counterfactual explanations have recently been brought to light as a potentially crucial response to obtaining human-understandable explanations from predictive models in Explainable Artificial Intelligence (XAI). Despite the fact that various counterfactual algorithms have been proposed, the state of the art research still lacks standardised proto...

There is a growing need to assist radiologists in performing X-ray readings and diagnoses fast, comfortably, and effectively. As radiologists strive to maximize productivity, it is essential to consider the impact of reading rooms in interpreting complex examinations and ensure that higher volume and reporting speeds do not compromise patient outco...

Laparoscopic surgery has the advantage of avoiding large open incisions and thereby decreasing blood loss, pain, and discomfort to patients. However, on the other side, it is hampered by restricted workspace, ambiguous communication, and surgeon fatigue caused by non-ergonomic head positioning. We aimed to identify critical problems and suggest des...

Immersive Colonography allows medical professionals to navigate inside the intricate tubular geometries of subject-specific 3D colon images using Virtual Reality displays. Typically, camera travel is performed via Fly-Through or Fly-Over techniques that enable semi-automatic traveling through a constrained, well-defined path at user-controlled spee...

Immersive colonography allows medical professionals to navigate inside the intricate tubular geometries of subject-specific 3D colon images using Virtual Reality displays. Typically, camera travel is performed via Fly-Through or Fly-Over techniques that enable semi-automatic traveling through a constrained, well-defined path at user-controlled spee...

Digital anatomy has emerged as an essential subfield of anatomy that processes the human body with the help of a computer. 3D reconstruction tools have been developed over the years, being always complementary to cadaver dissection. Teachers' main goal is to provide a greater understanding of the spatial structures of the body and its internal orga...

The pandemic brought about an unprecedented number of virtual conferences, given the heavy restrictions on travel to in-person meetings. Despite all the advances in technology, people still complain about virtual events. There is Zoom fatigue, confinement malaise, and a longing for personal social interactions. This paper discusses our experience o...

Deep learning models have achieved high performance across different domains, such as medical decision-making, autonomous vehicles, decision support systems, among many others. However, despite this success, the inner mechanisms of these models are opaque because their internal representations are too complex for a human to understand. This opacity...

Gamified learning aims to motivate students using game elements. Although gamification can enhance students’ enjoyment and engagement, it is unclear how different students behave in and interact with gamified contexts. To this end, we analyze how different students interact with a gamified course. We devised such an experimental course on Multimedi...

As the oldest medical craft, anatomy remains the core and foundational field of medicine. That is why anatomy is in perpetual advancement, thanks to the technical progress in exploring the human body through computer science and biomedical research. Knowledge of the human body is the basis of medicine. Classical cadaver dissection, the standard dis...

Computed Tomography (CT) is a commonly used imaging modality across a wide variety of diagnostic procedures (World Health Organisation 2017). By generating contiguous cross-sectional images of a body region, CT has the ability to represent valuable 3D data that enables professionals to easily identify, locate, and accurately describe anatomical lan...

3D reconstruction from anatomical slices permits anatomists to create three-dimensional depictions of real structures by tracing organs from sequences of cryosections. A wide variety of tools for 3D reconstruction from anatomical slices are becoming available for use in training and study. In this chapter, we present Anatomy Studio, a collaborative...

Anatomy studies the morphology and structure of organisms. The word originates from the Greekana-, up; and tome-, meaning cutting. As its name implies, anatomy relies heavily on dissection and studies the human body parts’ arrangement and interaction. Heir from a rich Greco-Roman tradition and background, Vesalius (1543) is arguably the father of m...

There has been a growing interest in model-agnostic methods that can make deep learning models more transparent and explainable to a user. Some researchers recently argued that for a machine to achieve a certain degree of human-level explainability, this machine needs to provide human causally understandable explanations, also known as causability....

Head-Mounted Virtual reality (VR) systems provide full-immersive experiences to users and completely isolate them from the outside world, placing them in unsafe situations. Existing research proposed different alert-based solutions to address this. Our work builds on these studies on notification systems for VR environments from a different perspec...

Corporate training relates to employees acquiring essential skills to operate equipment or effectively performing required tasks both competently and safely. Unlike formal education, training can be incorporated into the task workflow and performed during working hours. Increasingly, organizations adopt different technologies to develop both indivi...

Studies indicate that Augmented Reality (AR) is quickly becoming an effective technology for training programs. This systematic literature review (SLR) aims to screen works published on AR for corporate training. We describe AR training applications, discuss current challenges, literature gaps, opportunities, and tendencies of corporate AR solution...

This book offers readers fresh insights on applying Extended Reality to Digital Anatomy, a novel emerging discipline. Indeed, the way professors teach anatomy in classrooms is changing rapidly as novel technology-based approaches become ever more accessible. Recent studies show that Virtual (VR), Augmented (AR), and Mixed-Reality (MR) can improve b...

Immersive Colonography allows medical professionals to navigate inside the intricate tubular geometries of subject-specific 3D colon images using Virtual Reality displays. Typically, camera travel is performed via Fly-Through or Fly-Over techniques that enable semi-automatic traveling through a constrained, well-defined path at user controlled spee...

In mass customization, software configurators enable novice end-users to design customized products and services according to their needs and preferences. However, traditional configurators hardly provide an engaging experience while avoiding the burden of choice. We propose a Design Participation Model to facilitate navigating the design space, ba...

Understanding the morphological features characterizing a normal femoral head and acetabular cavity is critical for a more comprehensive and updated definition of hip anatomy. Based on anatomical observations, MacConaill introduced the notion that spheroidal articular surfaces are better represented by ovoidal shapes, in comparison with the still v...

New technologies allow ordinary people to access Virtual Reality at affordable prices in their homes. One of the most important tasks when interacting with immersive Virtual Reality is to navigate the virtual environments (VEs). Arguably, the best methods to accomplish this use of direct control interfaces. Among those, natural walking (NW) makes f...

Common natural walking techniques for navigating in virtual environments feature constraints that make it difficult to use those methods in cramped home environments. Indeed, natural walking requires unobstructed and open space, to allow users to roam around without fear of stumbling on obstacles while immersed in a virtual world. In this work, we...

Augmented reality (AR) allows computer generated graphics to be overlaid in images or video captured by a camera in real time. This technology is often used to enhance perception by providing extra information or simply by enriching the experience of the user. AR offers a significant potential in many applications such as industrial, medical, educa...

Face-to-face telepresence promotes the sense of ”being there” and can improve collaboration by allowing immediate understanding of remote people’s nonverbal cues. Several approaches successfully explored interactions with 2D content using a see-through whiteboard metaphor. However, with 3D content, there is a decrease in awareness due to ambiguitie...

Modern volumetric projection-based telepresence approaches are capable of providing realistic full-size virtual representations of remote people. Interacting with full-size people may not be desirable due to the spatial constraints of the physical environment, application context, or display technology. However, the miniaturization of remote people...

Extensive research has been applied to discover new techniques and methods to model protein-ligand interactions. In particular, considerable efforts focused on identifying candidate binding sites, which quite often are active sites that correspond to protein pockets or cavities. Thus, these cavities play an important role in molecular docking. Howe...

Foot input has been proposed to support hand gestures in many interactive contexts, however, little attention has been given con-tactless 3D object manipulation. This is important since many applications , namely sterile surgical theaters require contactless operation. However, relying solely on hand gestures makes it difficult to specify precise i...

A Interação Homem-Computador (IHC) tem sido mencionada como uma das áreas mais importantes para a integração entre seres humanos e as diferentes formas de tecnologias cotidianas. A IHC desempenhou um papel central na evolução das tecnologias computacionais, a interface gráfica do usuário, por exemplo, teve um impacto significativo na popularização...

We are living in a post-WIMP world. Indeed, more and more users access information, communicate and operate mobile information appliances foregoing the still common mouse and keyboard of yonder. However, no matter how powerful or elegant the new mobile devices are, it is the user interface that ultimately governs how successful new devices or syste...

Face-to-face telepresence promotes the sense of "being there" and can improve collaboration by allowing immediate understanding of remote people's nonverbal cues. Several approaches successfully explored interactions with 2D content using a see-through whiteboard metaphor. However, with 3D content, there is a decrease in awareness due to ambiguitie...

3D reconstruction from anatomical slices allows anatomists to create three dimensional depictions of real structures by tracing organs from sequences of cryosections. However, conventional user interfaces rely on single-user experiences and mouse-based input to create content for education or training purposes. In this work, we present Anatomy Stud...

Feet input can support mid-air hand gestures for touchless medical image manipulation to prevent unintended activations, especially in sterile contexts. However, foot interaction has yet to be investigated in dental settings. In this paper, we conducted a mixed methods research study with medical dentistry professionals. To this end, we developed a...

When engaged in communication, people often rely on point- ing gestures to refer to out-of-reach content. However, observers frequently misinterpret the target of a pointing gesture. Previous research suggests that to perform a pointing gesture, people place the index finger on or close to a line connecting the eye to the referent, while observers...

Introduction: The growing interest of Augmented Reality (AR) together with the renaissance of Virtual Reality (VR) opened new possibilities to redesign how professionals interact with medical images. Several medical specialties already rely on 2D and 3D image data for diagnosis, surgical planning, surgical navigation, medical education and patient-...

Introduction: Most medical specialties rely on 3D image data either for diagnosis, surgical planning, surgical navigation, education or patient-clinician communication. Surprisingly, conventional medical image workstations do not promote proper visual insight nor visual collaboration between professionals. Redesigning conventional applications into...

Locomotion in virtual environments is currently a difficult and unnatural task to perform. Normally, researchers tend to devise ground-floor-based metaphors, to constrain the degrees of freedom (DoFs) during motion. These restrictions enable interactions that accurately emulate human gait to provide high interaction fidelity. However, flying allows...

Despite its marked success in recent years, it is still not clear how Virtual Reality (VR) can assist architects at the early stages of ideation and design. In this paper, we approach VR to build and explore maquettes at different scales in early design stages. To this end we developed a VR environment where user interactions are supported by untet...