Oliver Amft’s research while affiliated with University of Freiburg and other places

Purpose We introduce a multimodel, real-time semantic segmentation and tracking approach for Augmented Reality (AR)-guided open liver surgery. Our approach leverages foundation models and scene-aware re-prompting strategies to balance segmentation accuracy and inference time as required for real-time AR-assisted surgery applications. Methods Our approach integrates a domain-specific RGBD model (ESANet), a foundation model for semantic segmentation (SAM), and a semi-supervised video object segmentation model (DeAOT). Models were combined in an auto-promptable pipeline with a scene-aware re-prompting algorithm that adapts to surgical scene changes. We evaluated our approach on intraoperative RGBD videos from 10 open liver surgeries using a head-mounted AR device. Segmentation accuracy (IoU), temporal resolution (FPS), and the impact of re-prompting strategies were analyzed. Comparisons to individual models were performed. Results Our multimodel approach achieved a median IoU of 71% at 13.2 FPS without re-prompting. Performance of our multimodel approach surpasses that of individual models, yielding better segmentation accuracy than ESANet and better temporal resolution compared to SAM. Our scene-aware re-prompting method reaches the DeAOT performance, with an IoU of 74.7% at 11.5 FPS, even when the DeAOT model uses an ideal reference frame. Conclusion Our scene-aware re-prompting strategy provides a trade-off between segmentation accuracy and temporal resolution, thus addressing the requirements of real-time AR-guided open liver surgery. The integration of complementary models resulted in robust and accurate segmentation in a complex, real-world surgical settings.

Introduction Inflammatory bowel disorders may result in abnormal Bowel Sound (BS) characteristics during auscultation. We employ pattern spotting to detect rare bowel BS events in continuous abdominal recordings using a smart T-shirt with embedded miniaturised microphones. Subsequently, we investigate the clinical relevance of BS spotting in a classification task to distinguish patients diagnosed with inflammatory bowel disease (IBD) and healthy controls. Methods Abdominal recordings were obtained from 24 patients with IBD with varying disease activity and 21 healthy controls across different digestive phases. In total, approximately 281 h of audio data were inspected by expert raters and thereof 136 h were manually annotated for BS events. A deep-learning-based audio pattern spotting algorithm was trained to retrieve BS events. Subsequently, features were extracted around detected BS events and a Gradient Boosting Classifier was trained to classify patients with IBD vs. healthy controls. We further explored classification window size, feature relevance, and the link between BS-based IBD classification performance and IBD activity. Results Stratified group K-fold cross-validation experiments yielded a mean area under the receiver operating characteristic curve ≥0.83 regardless of whether BS were manually annotated or detected by the BS spotting algorithm. Discussion Automated BS retrieval and our BS event classification approach have the potential to support diagnosis and treatment of patients with IBD.

We disentangle the efficacy of individual non-pharmaceutical interventions (NPIs), including digital contact tracing (DCT), with a novel behaviour-driven agent-based model (ABM) to inform ongoing pandemic preparedness efforts. Our model’s Zeitgeber architecture delineates contextual characteristics, including daytime, daily routines, locations, and activities. Our method determines each agent’s current location and behaviour in a realistic environment under the restrictions of NPIs. We model viral load transfer between agents from contact duration, distance, and the infected agent’s infectiousness level. We examine the effects of DCT adoption, adherence, and compliance, both individually and combined with other NPIs, on key pandemic indicators, thus providing novel insight into infection dynamics. DCT combined with other NPIs reduces the total infections up to 52% for realistic behaviour, whereas DCT alone yielded a 43% reduction. Surprisingly however, some NPI combinations do not improve pandemic parameters. Our approach offers fine-grained analysis capabilities on the effectiveness of NPI combinations that cannot be obtained in human studies due to confounding effects. Thus our approach can inform future pandemic control efforts and prioritisation in pandemic preparedness.

We investigate the impact of melanin concentration on the accuracy of oxygen saturation SpO ₂ estimation using reflective photopletsymography (PPG) measurements. Monte Carlo (MC) simulations were used to simulate photon-tissue interaction and detection of light for systole and diastole. We model a reflective pulse oximeter and analyse the perfusion index ( PI ), ratio-of-ratios RoR , and SpO ₂ depending on skin colour. We derive calibration models tailored to specific melanin concentrations ranging from 2.55% to 30.5%, as well as a generalised population model, to study reflective pulse oximeter performance. While current pulse oximeters are often calibrated for low melanin concentrations, our results show that calibrating for an appropriate skin colour range, substantially increases pulse oximeter performance. Skin colour-adapted calibration reduced the root mean square error A_rms compared to a generalised population model by an average of 44%. SpO ₂ estimation error was minimal at a distance of 4mm to 5mm between light source and photodetector of a reflective pulse oximeter. We conclude that skin colour adapted calibration should be applied to make reflective pulse oximeter similarly applicable to any human, independent of pigmentation.

Goal: We investigate the effect of source-detector geometry, including distance and angle, on the reflective photoplethysmography (PPG) signal. Methods: A porcine skin phantom was used for laboratory measurements and replicated by Monte Carlo simulations. Variations in sensor geometry were analysed. Results: Laboratory measurements and Monte Carlo simulations showed agreement for various geometry settings. With decreasing negative sensor angle, the differential path length factor and the average maximum penetration depth increases. Conclusions: Our analyses highlight the influence of sourcedetector geometry on the PPG DC signal. Based on our analysis of penetration depth and optical path length, the geometry effects can be transferred to the PPG AC signal too. MC simulations provide an important tool to optimise PPG performance.

We propose and analyze a schema to harmonize data and metadata encoding for wearable and ubiquitous devices based on the Fast Healthcare Interoperability Resources standard. Currently, the lack of dataset encoding standards contributes to interpretation errors and proprietary implementations. Our proposed schema describes sensor devices, assessment data, and their characteristics. We develop and evaluate our schema regarding usability, common metric support, FAIRness, and privacy, using four stress monitoring datasets.

Background Current health recommender systems lack interactivity that relates to the current situation. Methods We designed and implemented an intervention study for obese patients that incorporates context information obtained from smartphone and smartwatch sensors, gamification, as well as joint goal management of patients and health coaches. We developed a health behaviour recommendation system comprising of a smartphone application, cloud platform for data management, and a data dashboard for coaches. Results We conducted a three months long study and analysed data from eight patients, focusing on system function, patient adherence, satisfaction and overall impact of the proposed system on changing health-related habits. Along with data analysis, we also provide patient feedback collected during interview round after the end of the study. Conclusion Patients could successfully implement the goals using the EghiFit app. Challenges regarding the data collection, recommendation synthesis, and patient engagement persist. Furthermore, reliable sensor data processing on current smartphone platforms is difficult due to system restrictions. Future research should further integrate sensor data, gaming, and health behaviour intervention design using smart devices.

This workshop brings researchers together to discuss and explore how artificial intelligence (AI) can be used to improve general health. During our workshop at the MuC conference, we will focus on three main areas: developing ethical AI health recommendations, exploring how smart technologies in our homes can influence our health habits, and understanding how different types of feedback can change our health behaviors. The workshop aims to be a space where various research areas meet, encouraging a shared understanding and creating new ways to use AI to encourage healthy living. By focusing on real-world applications of AI and digital twins, we seek to guide our discussions toward strategies that have a direct and positive impact on individual and societal health.

This paper presents findings from our implementation of a context aware health guidance system for obese individuals, with a focus on smartphone- and smartwatch-based health monitoring and participant adherence. To aid participants in weight loss, our system utilizes data from wearables and smartphones, integrating nutrition tracking and gamification elements into a smartphone application and a web-based health dashboard for health coaches. Eight participants completed a 120-day field study to evaluate the system and examine user adherence to health monitoring and the effectiveness of gamification in a weight loss program. Data on steps, sleep, heart rate, weather, and manually logged meals were collected. Adherence varied across data types, with step counts being the most consistently collected, while sleep and heart rate data were limited due to inconsistent smartwatch usage.