PosterPDF Available

A mobile augmented reality app for biomedical research and education

Authors:

Abstract

A mobile augmented reality application for biomedical research and education Augmented reality (AR) technology combines a real-world view with computer-generated assets. AR devices range from high-end wearable headsets to smartphone-based applications that can be used with or without inexpensive stereoscopic viewer accessories. Our research and others' work has shown the potential to dramatically change biomedical research, education, and clinical practice. We implemented a mobile application for 3D visualization of biological structures in AR using the MERGE Cube TM (Merge Labs, Inc.), a 2.75" foam cube. A smartphone camera detects markings on the outer surface of the cube onto which a hologram is projected, thus allowing the user to manipulate the virtual object with their hands. Our mobile app prototype uses the MERGE Cube software developer kit to display 3D models of biological structures, either on the flat smartphone screen or with a wearable smartphone accessory for stereoscopic viewing. 3D manipulation of these models in AR provides a novel and convenient method for researchers and students to communicate and learn about structural and functional features of biomolecules and viruses. Our alpha version is developed for the Apple iOS platform, and we are incorporating additional biomedical structures besides influenza virus models and interactive features. Demonstrations of the app thus far have received positive feedback (especially from hundreds of users including teachers, students and researchers during 2018 US Sceince and Engineering Festival), and more extensive user testing is underway with a small number of NIAID research staff. Our intention is to extend the app for Android OS capability and to make it freely available to the public. Additional information about Merge Labs products, including the MERGE Cube, smartphone goggles, and the MERGE VR/AR SDK can be found at https://mergevr.com. Special thanks to Dr. Barney Graham (NIAID Vaccine Research Center), for championing this technology and providing valuable feedback for current and future work. The digital file of the influenza virion is available for download at https://3dprint.nih.gov/discover/3dpx-000030. The model was created using UCSF Chimera software (https://www.rbvi.ucsf.edu/chimera). Chimera is developed by the Resource for Biocomputing, Visualization, and Informatics at the University of California, San Francisco. Pettersen et al., UCSF Chimera-a visualization system for exploratory research and analysis. Above, the same digital 3D model of an influenza virion created from cryo-electron microscopy data-is represented as a 3D print (top left), in our augmented reality app (right), and in virtual reality (bottom left). Left: Example of augmented reality showing a digital 3D heart model on top of a real-world view, captured with the user's smartphone. Right: the Microsoft HoloLens is an example of a more sophisticated AR technology that projects holographic images onto glasses or a visor. The MERGE Cube (© Merge Labs, Inc., www.mergecube.com) is a 4"x4" foam cube that is used to manipulate holographic objects using the camera of a smartphone or tablet device. A digital 3D image is mapped to unique markings on each face of the cube detected by the camera. As the user moves the cube, the digital object moves correspondingly, and will also shrink and grow as the cube is moved away and toward the device. The lightweight, 2.72-ounce cube is made a of soft, squishy foam that is attractive as a toy, making it an engaging tool for education. Based on our experience with 3D printing, and more recently with immersive virtual reality, we posited that a mobile AR app would be a convenient and more accesible means for experts and lay users to learn and communicate about molecular structures and other bioscientific content. Using the MERGE AR/VR software development kit (SDK) we have created a prototype AR app, compatible with any iOS device (iPhone, iPad and iPod Touch) with iOS 8 or newer for the iOS platform. Our development environment uses the following software:-MergeCube SDK v1.2.2-Unity2017.2-XCode 9.2-Vuforia SDK 7.2-Blender 2.79 Left: a user points an iPhone at a MERGE Cube using the "phone" mode. The "VR" mode creates a stereoscopic view (above right) that adds depth perception to the experience. It also leaves both of the user's hands free to manipulate the cube. MERGE Cube and MERGEVR Goggles shown at bottom right. The MERGE Cube app can be used with most other consumer-level, affordable, 3D smartphone goggles. Applications and features The MERGE Cube is low cost, efficient, learning tool for users of all ages. It was met with positive enthusiasm when we demonstrated the prototype at the USA Science and Engineering Festival in Washington, D.C., in April 2018. The convenience and portability of our app make it a potentially impactful tool for researchers to communicate their findings. The cube is an attractive alternative to 3D-printed models, which can be heavy or fragile, and the app also allows access to multiple 3D models. Current and future development includes interactive touch-screen or vision-guided annotations; selecting from a library of models; and pre-processing pipelines to prepare 3D model files. (NIAID/VRC) uses a 3D print an influenza receptor to explain the structure during a conference presentation. With our app, he can carry many models with him to a conference, and the smartphone view can be projected to a screen for large demonstrations.
Abstract
A mobile augmented reality application for biomedical research and education
Augmented reality (AR) technology combines a real-world view with
computer-generated assets. AR devices range from high-end wearable
headsets to smartphone-based applications that can be used with or
without inexpensive stereoscopic viewer accessories. Our research and
others' work has shown the potential to dramatically change biomedical
research, education, and clinical practice.
We implemented a mobile application for 3D visualization of biological
structures in AR using the MERGE CubeTM (Merge Labs, Inc.), a 2.75"
foam cube. A smartphone camera detects markings on the outer
surface of the cube onto which a hologram is projected, thus allowing
the user to manipulate the virtual object with their hands.
Our mobile app prototype uses the MERGE Cube software developer
kit to display 3D models of biological structures, either on the flat
smartphone screen or with a wearable smartphone accessory for
stereoscopic viewing. 3D manipulation of these models in AR provides a
novel and convenient method for researchers and students to
communicate and learn about structural and functional features of
biomolecules and viruses.
Our alpha version is developed for the Apple iOS platform, and we are
incorporating additional biomedical structures besides influenza virus
models and interactive features. Demonstrations of the app thus far
have received positive feedback (especially from hundreds of users
including teachers, students and researchers during 2018 US Sceince and
Engineering Festival), and more extensive user testing is underway with
a small number of NIAID research staff. Our intention is to extend the
app for Android OS capability and to make it freely available to the
public.
Kai Zhang, Phil Cruz, Meghan C. McCarthy, and Darrell E. Hurt
Bioinformatics and Computational Biosciences Branch, Office of Cyber Infrastructure and Computational Biology, National Institute of Allergy and Infectious Diseases (NIAID), NIH, Bethesda, MD 20892, USA
Additional information about Merge Labs products, including the MERGE Cube, smartphone goggles, and the MERGE VR/AR
SDK can be found at https://mergevr.com.
Special thanks to Dr. Barney Graham (NIAID Vaccine Research Center), for championing this technology and providing
valuable feedback for current and future work.
The digital file of the influenza virion is available for download at https://3dprint.nih.gov/discover/3dpx-000030. The model
was created using UCSF Chimera software (https://www.rbvi.ucsf.edu/chimera). Chimera is developed by the Resource for
Biocomputing, Visualization, and Informatics at the University of California, San Francisco. Pettersen et al., UCSF Chimera - a
visualization system for exploratory research and analysis. J Comput Chem. 2004 Oct 25. 25(13):1605-12.
This project has been funded in part with Federal funds from the National Institute of Allergy and Infectious Diseases, National Institutes of Health, under the NIAID BCBB Support Services Contract
HHSN316201300006W/HHSN27200002 . Mention of trade names, commercial products, or organizations does not imply endorsement by the U.S. Government.
bioviz@mail.nih.gov
Above, the same digital 3D model of an influenza virion created from
cryo-electron microscopy data - is represented as a 3D print (top left), in our
augmented reality app (right), and in virtual reality (bottom left). Source model
was created by Dr. Darrell Hurt (NIH/NIAID) based on data provided by Bernard
Heymann (NIH/NCI).
PHYSICAL VIRTUALAUGMENTED, MIXED
Augmented reality technology involves the overlay of digital information
onto a real-world view, essentially “augmenting” the user’s view of their
environment.
Augmented Reality
The Merge Cube
Left: Example of augmented reality
showing a digital 3D heart model on top
of a real-world view, captured with the
user’s smartphone. Right: the Microsoft
HoloLens is an example of a more
sophisticated AR technology that projects
holographic images onto glasses or a visor.
The MERGE Cube (© Merge Labs, Inc.,
www.mergecube.com) is a 4”x4” foam cube that is used
to manipulate holographic objects using the camera of a
smartphone or tablet device.
A digital 3D image is mapped to unique markings on
each face of the cube detected by the camera. As the
user moves the cube, the digital object moves
correspondingly, and will also shrink and grow as the
cube is moved away and toward the device.
The lightweight, 2.72-ounce cube is made a of soft,
squishy foam that is attractive as a toy, making it an
engaging tool for education. Dozens of apps are available
for iOS or Android.
The cube is available online and in stores (Amazon, Walmart, Target, etc.). MSRP is $14.99, but it can be
purchased for as low as $5.
Left: Marking on three faces of the MERGE Cube. Right:
The MERGE Cube foam is soft, but quickly returns to its
original shape.
A MERGE Cube app for scientific visualization
Based on our experience with
3D printing, and more recently
with immersive virtual reality,
we posited that a mobile AR
app would be a convenient and
more accesible means for
experts and lay users to learn
and communicate about
molecular structures and
other bioscientific content.
Using the MERGE AR/VR
software development kit
(SDK) we have created a
prototype AR app, compatible
with any iOS device (iPhone,
iPad and iPod Touch) with iOS
8 or newer for the iOS
platform.
Our development
environment uses the
following software:
- MergeCube SDK v1.2.2
- Unity2017.2
- XCode 9.2
- Vuforia SDK 7.2
- Blender 2.79
Left: a user points an iPhone at a MERGE Cube using the “phone” mode. The “VR” mode creates a stereoscopic view
(above right) that adds depth perception to the experience. It also leaves both of the user’s hands free to manipulate
the cube. MERGE Cube and MERGEVR Goggles shown at bottom right. The MERGE Cube app can be used with most
other consumer-level, affordable, 3D smartphone goggles.
Applications and features
The MERGE Cube is low cost, efficient, learning
tool for users of all ages. It was met with
positive enthusiasm when we demonstrated the
prototype at the USA Science and Engineering
Festival in Washington, D.C., in April 2018.
The convenience and portability of our app
make it a potentially impactful tool for
researchers to communicate their findings. The
cube is an attractive alternative to 3D-printed
models, which can be heavy or fragile, and the
app also allows access to multiple 3D models.
Current and future development includes
interactive touch-screen or vision-guided
annotations; selecting from a library of models;
and pre-processing pipelines to prepare 3D
model files.
Left: a young visitor to the USA Science and Engineering
Festival manipulate the cube with the smartphone inserted
into goggles. (Credit: USA Science & Engineering Festival
Sponsor Recap Report. https://bit.ly/2HKf2cy). Right: Dr.
Barney Graham (NIAID/VRC) uses a 3D print an influenza
receptor to explain the structure during a conference
presentation. With our app, he can carry many models
with him to a conference, and the smartphone view can be
projected to a screen for large demonstrations.
ResearchGate has not been able to resolve any citations for this publication.
ResearchGate has not been able to resolve any references for this publication.