A mobile augmented reality application for biomedical research and education
Augmented reality (AR) technology combines a real-world view with
computer-generated assets. AR devices range from high-end wearable
headsets to smartphone-based applications that can be used with or
without inexpensive stereoscopic viewer accessories. Our research and
others' work has shown the potential to dramatically change biomedical
research, education, and clinical practice.
We implemented a mobile application for 3D visualization of biological
structures in AR using the MERGE CubeTM (Merge Labs, Inc.), a 2.75"
foam cube. A smartphone camera detects markings on the outer
surface of the cube onto which a hologram is projected, thus allowing
the user to manipulate the virtual object with their hands.
Our mobile app prototype uses the MERGE Cube software developer
kit to display 3D models of biological structures, either on the flat
smartphone screen or with a wearable smartphone accessory for
stereoscopic viewing. 3D manipulation of these models in AR provides a
novel and convenient method for researchers and students to
communicate and learn about structural and functional features of
biomolecules and viruses.
Our alpha version is developed for the Apple iOS platform, and we are
incorporating additional biomedical structures besides influenza virus
models and interactive features. Demonstrations of the app thus far
have received positive feedback (especially from hundreds of users
including teachers, students and researchers during 2018 US Sceince and
Engineering Festival), and more extensive user testing is underway with
a small number of NIAID research staff. Our intention is to extend the
app for Android OS capability and to make it freely available to the
Kai Zhang, Phil Cruz, Meghan C. McCarthy, and Darrell E. Hurt
Bioinformatics and Computational Biosciences Branch, Office of Cyber Infrastructure and Computational Biology, National Institute of Allergy and Infectious Diseases (NIAID), NIH, Bethesda, MD 20892, USA
Additional information about Merge Labs products, including the MERGE Cube, smartphone goggles, and the MERGE VR/AR
SDK can be found at https://mergevr.com.
Special thanks to Dr. Barney Graham (NIAID Vaccine Research Center), for championing this technology and providing
valuable feedback for current and future work.
The digital file of the influenza virion is available for download at https://3dprint.nih.gov/discover/3dpx-000030. The model
was created using UCSF Chimera software (https://www.rbvi.ucsf.edu/chimera). Chimera is developed by the Resource for
Biocomputing, Visualization, and Informatics at the University of California, San Francisco. Pettersen et al., UCSF Chimera - a
visualization system for exploratory research and analysis. J Comput Chem. 2004 Oct 25. 25(13):1605-12.
This project has been funded in part with Federal funds from the National Institute of Allergy and Infectious Diseases, National Institutes of Health, under the NIAID BCBB Support Services Contract
HHSN316201300006W/HHSN27200002 . Mention of trade names, commercial products, or organizations does not imply endorsement by the U.S. Government.
Above, the same digital 3D model of an influenza virion created from
cryo-electron microscopy data - is represented as a 3D print (top left), in our
augmented reality app (right), and in virtual reality (bottom left). Source model
was created by Dr. Darrell Hurt (NIH/NIAID) based on data provided by Bernard
PHYSICAL VIRTUALAUGMENTED, MIXED
Augmented reality technology involves the overlay of digital information
onto a real-world view, essentially “augmenting” the user’s view of their
The Merge Cube
Left: Example of augmented reality
showing a digital 3D heart model on top
of a real-world view, captured with the
user’s smartphone. Right: the Microsoft
HoloLens is an example of a more
sophisticated AR technology that projects
holographic images onto glasses or a visor.
The MERGE Cube (© Merge Labs, Inc.,
www.mergecube.com) is a 4”x4” foam cube that is used
to manipulate holographic objects using the camera of a
smartphone or tablet device.
A digital 3D image is mapped to unique markings on
each face of the cube detected by the camera. As the
user moves the cube, the digital object moves
correspondingly, and will also shrink and grow as the
cube is moved away and toward the device.
The lightweight, 2.72-ounce cube is made a of soft,
squishy foam that is attractive as a toy, making it an
engaging tool for education. Dozens of apps are available
for iOS or Android.
The cube is available online and in stores (Amazon, Walmart, Target, etc.). MSRP is $14.99, but it can be
purchased for as low as $5.
Left: Marking on three faces of the MERGE Cube. Right:
The MERGE Cube foam is soft, but quickly returns to its
A MERGE Cube app for scientific visualization
Based on our experience with
3D printing, and more recently
with immersive virtual reality,
we posited that a mobile AR
app would be a convenient and
more accesible means for
experts and lay users to learn
and communicate about
molecular structures and
other bioscientific content.
Using the MERGE AR/VR
software development kit
(SDK) we have created a
prototype AR app, compatible
with any iOS device (iPhone,
iPad and iPod Touch) with iOS
8 or newer for the iOS
environment uses the
- MergeCube SDK v1.2.2
- XCode 9.2
- Vuforia SDK 7.2
- Blender 2.79
Left: a user points an iPhone at a MERGE Cube using the “phone” mode. The “VR” mode creates a stereoscopic view
(above right) that adds depth perception to the experience. It also leaves both of the user’s hands free to manipulate
the cube. MERGE Cube and MERGEVR Goggles shown at bottom right. The MERGE Cube app can be used with most
other consumer-level, affordable, 3D smartphone goggles.
Applications and features
The MERGE Cube is low cost, efficient, learning
tool for users of all ages. It was met with
positive enthusiasm when we demonstrated the
prototype at the USA Science and Engineering
Festival in Washington, D.C., in April 2018.
The convenience and portability of our app
make it a potentially impactful tool for
researchers to communicate their findings. The
cube is an attractive alternative to 3D-printed
models, which can be heavy or fragile, and the
app also allows access to multiple 3D models.
Current and future development includes
interactive touch-screen or vision-guided
annotations; selecting from a library of models;
and pre-processing pipelines to prepare 3D
Left: a young visitor to the USA Science and Engineering
Festival manipulate the cube with the smartphone inserted
into goggles. (Credit: USA Science & Engineering Festival
Sponsor Recap Report. https://bit.ly/2HKf2cy). Right: Dr.
Barney Graham (NIAID/VRC) uses a 3D print an influenza
receptor to explain the structure during a conference
presentation. With our app, he can carry many models
with him to a conference, and the smartphone view can be
projected to a screen for large demonstrations.