MethodPDF Available

A Medical Visualization Framework and Pipeline for Holographic MRI

Authors:

Abstract and Figures

In this work we present a visualization pipeline to express a series of magnetic resonance imaging (MRI) volume data, as a full colour hologram. Our pipeline highlights salient structure information which reside within the data without extensive processing of the raw MRI, utilizing its full spatial and temporal characteristics. We present a unique approach to MRI visualization highlighting the importance of using volumetric colormaps and the physical characteristics of holography in the medical field to easily reveal depth information. We employ artist-driven colour techniques which make use of luminance and saturation rather than the current standard of contrast of hues. This research provides a brief overview of the current spatial displays in medical imaging application and describe both the current methods of processing (MRI dataset → model construction → computer graphic rendering → hologram encoding) and our proposed pipeline (MRI dataset → colour mapping → hologram encoding). We present the results of our pipeline and highlight various structures within the data without the use of sophisticated segmentation techniques or specialized software. The research is the first of two stages in our quest to provide tactile three dimensional images in the hands of the medical imaging community.
Content may be subject to copyright.
A medical visualization framework and pipeline for
holographic MRI
Michael Pagea, Marcus A. Gordona, Mario Garingoa, Adriana Menghia, Jawa El Khasha, and
Dr. Trevor McKeeb
aPrototype Holographics for Art-Science Exploration (PHASE) Lab, OCAD University,
Toronto, Canada
bUniversity Health Network (UHN) Research, Toronto, Canada
ABSTRACT
In this work we present a visualization pipeline to ex-
press a series of magnetic resonance imaging (MRI) vol-
umetric data, as a full colour hologram. Our pipeline
highlights salient structure information which reside
within the data without extensive processing of the raw
MRI, utilizing its full spatial and temporal character-
istics. We present a unique approach to MRI visual-
ization highlighting the importance of using volumetric
colormaps and the physical characteristics of hologra-
phy in the medical field to easily reveal depth informa-
tion. We employ artist-driven colour techniques which
make use of luminance and saturation rather than the
current standard of contrast of hues. This report pro-
vides a brief overview of the current spatial displays
in medical imaging application and describes both the
existing method for image processing (MRI dataset
model construction computer graphic rendering
hologram encoding) and our proposed pipeline (MRI
dataset colour mapping hologram encoding). We
present the results of our pipeline and highlight how
various structures within the data can be observed
without the use of sophisticated segmentation tech-
niques or specialized software. These results provide
an initial methodological framework necessary for fur-
thering our quest to provide tactile three dimensional
holographic images in the hands of the medical imaging
community.
Keywords: medical holography, 3D visualization, dig-
ital holography, volume rendering, MRI, colormaps,
segmentation, physically based rendering
1. INTRODUCTION
In medicine, doctors rely on visualization tools to
display tumor or any other lesions, in a clear man-
ner such that the spatial relationship to anatomical
structures of the brain are perceived. Typically,
Further author information: (Send correspondence to
Marcus A. Gordon)
E-mail: magfoto@yorku.ca
they use magnetic resonance images (MRI) to obtain
volumetric data and display them onto a monitor.
They manipulate the three dimensional data, inherent
to MRI, as two dimensional data through a monitor.
These visualization tools often show various angles of
the data so that the physician can infer the relative
position, orientation, shape and relevant anatomical
cues to disambiguate the spatial relationship between
the tumor and the rest of the brain structure.
We believe that viewing 3D structured data is best
viewed using true 3D displays such as holograms. As
such, we hypothesize that it can be more effective in
representing tumor shape and position especially in
situations when the shape is abnormal and the tissue
is statistically similar to its surrounding. Furthermore,
holograms have an additional advantage over other
3D displays in that no special viewing aids are needed
such as special glasses or a head worn display such
as a VR headset. In addition, because holograms
can be made in colour, we can utilize different color
mapping techniques to highlight various regions in
the MRI without manipulating the data, which is a
necessity in the medical field.1Furthermore, in the
past, depth was used as a colour parameter2but using
holographic depth is inherent, so we can use the raw
data as parameters for colour instead. Literature
has established that with the appropriate colour
mapping techniques, colormaps play an important role
in visualization because they are able to improve the
efficiency and effectiveness of data perception which in
turn allow for more insights into the data.3
The main focus and contribution of this research is
to highlight the importance of colour as well as the
visualization medium to optimally view medical volu-
metric data such as MRIs, and provide an alternative
visualization pipeline to viewing them.
2. BACKGROUND
This research stems from previous work in developing
direct-write holography prints of a voxelized brain
model, to develop awareness in brain research through
the exhibition of such works. While our interest and
advocacy of medical holography grew, members of
our research team presented holographic works to the
Ontario Medical Association (OMA) in 2016, and in
the International Symposium for Display Holography
in St-Petersberg, Russia in 2015 and at the University
of Aveiro in Aveiro City, Portugal in 2018.
The research also adds to the work of ColorMoves,4
where colormap applications in scientific visualization
were explored with use cases in climate science, neu-
tron spectroscopy, and asteroid ocean impacts. In the
writeup that follows, we further this exploration with
medical imaging, with a focus on applying divergent
and structured colormaps to volume data.
3. THEORY
3.1 Colour Mapping, an Artist-Driven
Colour Method Approach
In the literature as well as in most scientific visual-
ization programs, colour has been proven to be an
effective tool. However, no single colormap is optimized
for all science domains. Current default colour schemes
such as the rainbow, jet, gray-scale and cool to warm
often obscure data and fail to solve the occlusion
relationships in volumetric rendering of data.5There
have been work in this field to combat this issue and
user-study-based, rule-based and data-driven methods
have been explored. However, we believe that the
work done by researchers at Los Alamos National
Laboratory is the most effective method in solving
this occlusion problem.4We extend their work into
the 3D medical field by applying two principle from
artistic colour theory: maximizing values contrast and
avoiding simultaneity of colour.
Human ability to focus attention and discriminate
detail is governed by the type and degree of contrast
and not by specific hues.6In our case study of using a
human brain and tumor, it is essential for the tumor
to be apparent and salient in the scene.
Figure 1. Linear Colormap from ColorMoves.
Figure 2. Divergent Colormap from ColorMoves.
Figure 3. Structured Colormap from ColorMoves.
One major pitfall of the current colour schemes is
the visual vibration caused by the abutting saturated
hues of the colourmaps. As a result our mapping
system will focus more on the allocation of contrast,
specifically luminance and saturation such as the ones
in Figures 1,2, and 3.
3.2 Volumetric Rendering
The current approach that our group has been using
to generate medical holograms is first constructing
a model using segmentation. For this we employ
an intuitive approach of pixel classification using
multi-thresholding7techniques in combination with
region growing approaches.8Following this step we
perform computer graphic rendering, and hologram
encoding. There are three main issues with this
pipeline. The first, is that it is an non-automatic
method which involves human intervention, second, is
that it is computationally expensive, and the third is
that it generates occlusion between the geometries in
the scene.
Occlusion is an important and powerful cue to scene
layout, and based on psychology studies it usually
caries the precept even in the presence of conflicting
information.9Analogous to colour, the rendering
approach should also report the occlusion relationship
between objects within the scene. In this diagnostic
application it is important not to hide information of
potential interest. As a result, the semi-transparent
and non-occluding nature of volumetric rendering may
be appropriate for some medical diagnostic applica-
tions.
3.3 Holograms
Holograms (digital optical holography) represent a
means of auto-stereoscopic, spatial recording of data.
The latest generation of full-colour direct-write holog-
raphywhich boasts higher resolution, greater angle of
Direct-write holograms are comprised of a multitude
(typically 10s of thousands) of hogels or holographic pixels.
These hogels are written directly on the recording medium
as an array of holograms, each “projecting” out their own
version of RGB along with phase information. The smaller
view and full parallax has opened up a new chapter in
holographic medical imaging.10
Although the images created are three dimensional,
the holographic printer works by virtually photograph-
ing two dimensional image sequences. These sequences
(or position slides) are put together as a stack, and
fed to the digital holographic printer. Using a spatial
light modulator, the printer invokes a pixel swapping
algorithm that connects the various points in 2D space,
adding relational z-depth cue information thus forming
a three dimensional recording.11
image plane
a1
c
d
e
b
a
ø
b1
Figure 4. Virtual camera model.
Creating an frame stack with our proposed virtual
camera model(see Figure 4) is necessary to feed
information to the holographic printer correctly,
and is a crucial step in generating the overall scene
design of the hologram viewed by the observer. As
a result, we explore two image processing avenues
using a set of 16-bit color images and a set of higher
color depth images to compare aesthetic differences.
32-bit high dynamic range (HDR) images are of
interest to investigate further in formats such as
OpenEXR, as they would suggest being much brighter
and thus contribute to a brighter hologram.12 The
current DICOM GSDF (grayscale standard display
function) specification suggests a grayscale image
the hogels and the closer they are together, the higher the
resolution.
Diagram depicts a virtual camera model based on three
step calculations of (1) linear transformation, (2) image
plane pixels, and (3) hogel spatial position
format at 16-bit depth,13 making our MRI dataset
colour mapping hologram encoding, plausible
within an HDR context from beginning to end in
our proposed pipeline. It is along these lines that
have inspired our team to take this path in the research.
4. METHODS
4.1 Pipeline Experiments
4.1.1 Overview
We created a virtual linear camera setup to match
the settings of various next-generation direct-write
holography printers, to facilitate hologram encoding.
We set out to test traditional segmentation methods,
converting the MRI data to three dimensional form,
and creating colormaps to apply to the data.
We looked to answering the following questions to
plan our experiments: (1) How do current image seg-
mentation techniques translate into three-dimensional
geometry? (2) What are the implications of using a
linear image processing workflow and high dynamic
range images14,15 in our pipeline? (3) What methods
can be used to apply colormaps to a volume that is
effective for hologram viewing?
4.1.2 Experiments
Hologram Encoding Holograms are by nature,
narrow band recordings. This means that with the
use of super-bright LEDs as the illuminating source,
(similarly, narrow band devices) one can isolate the
RGB channels. Full colour in medical imaging, is
typically used to denote tissue density or to delineate
bone for example from soft tissue. This makes possible
the introduction of the red channel and animation,
the addition of the green channel, the introduction
of colour mixing components in yellow and finally
the introduction of the blue channel and additional
animation. Further experiments are being conducted
into the gesture recognition control over the above
parameters.
Learnings from VFX Our approach here uses a
texture-based volumetric rendering to generate the
views of the medical images.
Combining this approach with the results from
luminance distribution tests with colormaps, we create
a workflow of exploring volume data and representing
high quality views needed for analysis. 3D volume
segmentation techniques are further explored applying
luminance distribution in other ways such as using
nested colormaps3in our volume rendering for greater
lighting control, and shifting the order of image
processing tasks such as applying colourmaps to
independent slices versus the direct volume.
After traditional image segmentation was applied
to our brain tumour dataset, this was followed by a
series of volume rendering experiments derived for the
pairing of the two factors in which our research sug-
gests are fundamentally linked to improving cognition
in viewing images: maximizing values contrast and
avoiding simultaneity of colour.4
These experiments tested the application of various
colormaps to the MRI volume at multiple stages of the
pipeline to test for optimal viewing results. Previous
research literature suggests a significant change in
default colormpas in various scientific visualization
platforms from MATLAB to ParaView, moving away
from hue focused colormaps. Researchers at Los
Alamos National Laboratory have conducted extensive
explorations in this problem domain, showing case
studies of their recommmended improvements in
climate and ocean science.4
Having chosen Blender16 mainly as a visualization
tool for our pipeline prototype, this decision was in
part due to its scripting ability in Python. Tests
were performed in Blender Python to process the
MRI dataset in generating an image stack for volume
rendering, automating the application of the colormaps
to the MRI volume, and scripting the linear camera
setup to generate specific frame stack specifications
for holographic printing. Similarly, OpenImageIO
(OIIO)17 an image input/output utility, is used for
both its quick shell and Python scripting abilities. In
our pipeline, OIIO processes and creates our image
stacks, and encodes the application of a defined
colormap onto the MRI slices.
Volume rendering is about the travel of light through
media within the boundaries of a 3D region. Blender
does this via a physically-based model, representing
the interactions of light realistically, making this our
primary prototyping environment for our scientific
visualization1819 and volume rendering experiments.
5. RESULTS
5.1 Geometry vs Volume
5.1.1 Geometry and Image Segmentation
Segmentation
There are several different approaches that have
been found throughout the literature,20 which all stem
from the same set of core ideas. In this work we
explored both automatic and semi-automatic methods.
In our data set there were 181 slices, each containing
an image with the dimensions 256x256. For each
slice an automatic and semi-automatic segmentation
approach was used. For the automatic approach we
employed transfer leaning and 3D U-Net dense volu-
metric segmentation.21 However, data used by21 was
too different from the data we are using, so the transfer
learning did not give us the adequate segmentation
we wanted. Therefore, we relied on a more intuitive
approach of pixel classification using multithresholding
(MT) techniques22 in combination with region growing
approaches,23 followed by a marching cube24 algorithm
to generate geometry.
Results of the manual seed region growing techniques
and morphological filters can be seen in figure 5.
Figure 5. Top Row: Original MRI. Middle Row:White
matter mask. Bottom Row:Gray matter mask.
We applied this technique not only for the regular
brain but also for a brain which contains a tumor. The
following set of figures shows examples slices as well as
their resultant class labels and individual mask layers.
Even though we applied morphological filters, we
can still see some noise within the images, specifically
the speckle effects in the gray matter and some holes
in the white matter masks.
Each unique colour corresponds to a unique known
tissue type. By looking at various views you can get a
picture as to where the tumor maybe located. To create
contrast between each distinct region we proposed the
following colouring scheme.
Figure 6. Colour coded regions of the brain
Tissue R G B
White 7 150 204
Gray 204 204 204
Abnormal Tissue 203 119 17
Tumor 203 2 2
Table 1. RGB values of each of the tissue types.
Notice that each of the tissue types can be clearly
distinguish from each other.
Geometry Generation From a far it may seem
that the models are what we expect, but looking at
the smaller object such as the tumor and the abnormal
tissue, it is clear that the resolution of the models are
directly correlated with the number of slices within the
DICOM file. The marching cube algorithm generates
these step wise artifacts which are the depth of the each
slice within the MRI. Because we can’t use smoothing
techniques, the only remedy to this is to increase the
number of slices. This will increase the computational
cost and time to create a model for printing. For most
use cases we think that this is okay in the long run for
holographic prints. But for regular clinical practice like
in the operating room or emergency care this is not
feasible. Nevertheless we prove that it is possible to
perform semi-automatic segmentation in conjunction
with a marching cube algorithm to develop geometry.
5.1.2 Volume and Colour
Our first rendering experiment was setup to create an
as-is volumetric rendering of the MRI data without
any colour mapping applied. The result of this was an
opaque plasticine-like form, and reveals inner structure
when sliced. Opaque views created a three dimensional
form in the viewing space that had defined markings
on the brain surface. In Figure 7our rendering sam-
ple depicts an applied red transmission colour in the
render.
Using OpenEXR images, three stacks were created
based on a divergent colormap, a linear-divergent
colormap, and a structured colormap created with
Colormoves. In Figure 8our three subjects have
strongly identical characteristics, regardless of their
disparate luminance properties. When testing different
colormaps applied to our MRI stack, we separated our
tests into two major processes (i.e. Process A. and
Process B.) where we compared different color ramp
setups (see Figure 9).
Transmission Colour R G B
Red 1.000 0.015 0.016
Yellow 1.000 0.646 0.165
Table 2. RGB floating point values of transmission colours
used in the experiments.
Notably, the difference test between A and B looked
at the relationship of luminance based color ramps and
full colour ones, and their impact on the volume data.
Process A. was originally considered the ideal approach
prior to running the experiments, however, Process B.
proved to be much more interesting lighting results
that were sometimes more revealing than Process A.
Slight opaque regions are revealed from the top
down in opaque slices. Translucent slices have a more
crystallized look and feel to the image when sliced,
however the TC regions are much more prominent.
Luminance is key to the ability to pinoint detail in
segmentation, and the single most important factor
in selecting colormaps in this process. Previous
research suggests that luminance is much stronger
Figure 7. Sample from first slice rendering result, using 16-
bit TIFF image stack.
with colormaps that have two or more interpolation
points, and as such has directed our preference towards
divergent colormaps and structure colormaps.
Figure 8. Visually identical renders with three different col-
ormaps applied.
The search for increased luminance has been the cri-
teria that has steered our research towards a focus on
linear workflow, and simple volumetric lighting. These
characteristics of our proposed pipeline are strongly
complemented by adding colormaps to the process at
either the input stage of MRI data processing or during
the volumetric rendering process. Both methods reveal
more insights from the data of the MRI, and create a
variety of options for integrating color and luminance
within the workflow.
5.2 Pipeline
The creation of images for this project occur in two
parts: (1) the making of an image stack for input to
the volume rendering pipeline, and (2) the making
of a frame stack for input to the holographic printer.
Achieving best results in the volume rendering process
suggests using a linear color space input to maintain
optimal luminance in post-processing.25 Additionally
important, is maintaining a precise number of image
frames to map out a three dimensional scene prior to
the pixel swapping that occurs in the printing process.
Process A.
Process B.
Figure 9. Color ramp node setups to apply colormaps to
volume data.
5.2.1 Image Stack Generation
To simplify the objectives of the first phase of this
research, we maintained a focus on an image type
that had 16-bit colour depth which is well known to
our VFX tools we chose to use in these experiments.
The image stack generation stage of the pipeline
requires a solution to address image file types used, file
conversion needs, and image resolution. We targeted
two reliable image file formats TIFF and OpenEXR,
both of which operate in 16-bit depth. A core image
stack was created by converting the original DICOM
file (.DCM) to an image sequence of 16-bit gray TIFF
files using ImageJ.26
5.2.2 Frame Stack Generation
Where the image sequence fed to the holographic
printer is an MPEG stream, the images begin life
as individual frames composed from a multitude of
angles. These frames run through a process defined as
“pixel swapping”, and the data in the 2D images come
together to synthesize the 3D image. All of the pixels
in the upper right corner of each of the 2D images are
used to form the hogels in the upper right corner of
the hologram.
5.2.3 Proposed Model
The colour mapping process was applied to the volume
data, and tests show that a divergent and/or structured
MRI Dataset
Generate
image stack
Apply colormap
Generate colour stack
Render volume
Generate colormap
Distribute slices
in Z-depth
Camera setup
Generate
frame stack
Figure 10. Master pipeline flow
colormap yield preferred results of highlighting image
detail.
As a result, our proposed pipeline consists of tasks
condensed into three major steps: (1) MRI data prepa-
ration, (2) colour mapping, and (3) hologram encoding.
The MRI data preparation involved the extraction of
MRI Dataset Generate
image stack
convert to 16-bit gray TIFF
Figure 11. DICOM slices are converted to TIFF.
TIFF16 Generate
image stack
(conversion) OpenEXR
Figure 12. TIFF16 stack converted to 16-bit OpenEXR.
a MRI DICOM or NIFTI medical file of 181 slices
for our brain, after which we applied segmentation to
isolate the tumour in the image data in traditional
DICOM GSDF grayscale. This breakdown suggests
the volumetric rendering to be automatic and allow for
user intervention at colour mapping stages.
5.3 Medical Data Review
Figure 13. Refined yellow transmission colour.
One the immediate considerations from this volume
rendering research is it’s similarity and dissimilarity to
the traditional segmentation process:
volume rendering introduces various lighting con-
trol and capability
segmentation describes tissue types via colour but
not inherent in volume rendering
the slices view of the volumetric renders provide
more visible form and structure than traditional
segmentation and geometry generation provides.
As a result, our partners in the medical community
confirm that the results of our experiments are visibly
more true to the medical data. Other advantages to
our approach that have been confirmed, include:
mimicing the natural way of viewing a three di-
mensional object
depth perception and navigation is more accus-
tomed in natural 3D
the slices view of the volumetric renders provide
more visible form and structure than traditional
segmentation and geometry generation provides.
The combination of the above results and observa-
tions makes for our research contribution. An addi-
tional highlight to the process is the application of col-
ormaps to more than a few slices at a time. This could
be a reduction of time spent classifying data, but more
notably is the fluidity of doing this in a visual manner
that comes natural to the subject. With this visualiza-
tion pipeline, the tactile prints proposed will also free
the user’s mind from additional work.
Figure 14. Structured colormap using Process B.
6. DISCUSSIONS
There are a myriad of factors to consider in the vol-
ume rendering including scatter, asymmetry, texture
density, emission and reflection. Our observations on
these properties led us to standardize parameters for
the majority of our experiments (see Table 3).
Shading Values
Transmission Yellow
Scattering 0.500
Assymetry 0.000
Emission 0.150
Reflection 0.750
Table 3. Consistent shading parameters used for the volume
rendering experiments.
The widening of our exploration of VFX tools
stemmed from our struggle with traditional scientific
visualization tools, which mostly did not appease to
our working methods. As our research was approached
in an art-science fashion, part of the dynamic of the
work is creative coding prototypes using various tech-
nologies. During this research, we have also explored
visual programming and real-time coding,27 and this
has now found its way into the next phase of our re-
search in kinetic holography.
Figure 15. Nested colormap using Process A.
For future work, we are currently preparing for the
second phase of this investigation and implement the
pipeline in other contexts. Physically based rendering
techniques learned for this research has also led us to
consider the various shading parameters of the volume
rendering process, and how it will contribute to the
tactile prints we make in the next phase.
ACKNOWLEDGMENTS
This medical holography research was funded by the
Ontario Centres of Excellence. We thank our Univer-
sity Health Network partner, Dr. Trevor McKee, Di-
rector of 3D4MD, Dr. Julielynn Wong MD, MPH, and
Dr. D’Arcy Little Chief, Diagnostic Imaging, Orillia
Soldiers Memorial Hospital.
REFERENCES
[1] Elgort, D., Wong, E., Hillenbrand, C., Lewin, J. S.,
and Duerk, J. L., “Adaptive imaging parameters
with mri,” (Jan. 5 2010). US Patent 7,643,864.
[2] Yuan, J., Chow, S., Yeung, D., and King, A., “A
five-colour colour-coded mapping method for dce-
mri analysis of head and neck tumours,” Clinical
radiology 67(3), 216–223 (2012).
[3] Samsel, F., Petersen, M., Geld, T., Abram, G.,
Wendelberger, J., and Ahrens, J., “Colormaps that
improve perception of high-resolution ocean data,”
in [Proceedings of the 33rd Annual ACM Con-
ference Extended Abstracts on Human Factors in
Computing Systems], 703–710, ACM (2015).
[4] Samsel, F., Klaassen, S., and Rogers, D. H., “Col-
ormoves: real-time interactive colormap construc-
tion for scientific visualization,” IEEE computer
graphics and applications 38(1), 20–29 (2018).
[5] van der Walt, S. and Smith, N., “mpl colormaps.”
https://bids.github.io/colormap/ (2016).
[6] Ware, C., “Color sequences for univariate maps:
Theory, experiments and principles,” IEEE Com-
puter Graphics and Applications 8(5), 41–49
(1988).
[7] Banerjee, S., Mitra, S., and Shankar, B. U., “Sin-
gle seed delineation of brain tumor using multi-
thresholding,” Information Sciences 330, 88–103
(2016).
[8] Wegli´nski, T. and Fabija´nska, A., “Brain tumor
segmentation from mri data sets using region grow-
ing approach,” in [Perspective Technologies and
Methods in MEMS Design], 185–188, IEEE (2011).
[9] Shipley, T. F. and Kellman, P. J., “Perception
of partly occluded objects and illusory figures:
Evidence for an identity hypothesis.,” Journal of
Experimental Psychology: Human Perception and
Performance 18(1), 106 (1992).
[10] Gentet, Y. and Gentet, P., “Chimera, a new
holoprinter technology combining low-power con-
tinuous lasers and fast printing,” Applied Op-
tics 58(34), G226–G230 (2019).
[11] Benton, S. A. and Bove Jr, V. M., [Holographic
imaging], John Wiley & Sons (2008).
[12] Kainz, F., Bogart, R., and Stanczyk, P., “Techni-
cal introduction to openexr,” Industrial light and
magic , 28 (2009).
[13] Choi, M., Wang, J., Albani, L., and Badano,
A., “60.2: Minimizing veiling glare in the high-
luminance-range visualization of medical images,”
SID Symposium Digest of Technical Papers 43(1),
816–819 (2012).
[14] Reinhard, E., Heidrich, W., Debevec, P., Pat-
tanaik, S., Ward, G., and Myszkowski, K., [High
dynamic range imaging: acquisition, display, and
image-based lighting], Morgan Kaufmann (2010).
[15] Hung, A. H., Liang, T., Sukerkar, P. A., and
Meade, T. J., “High dynamic range processing for
magnetic resonance imaging,” PLOS ONE 8, 1–11
(11 2013).
[16] Blender Foundation, “Blender.”
[17] Gritz, Larry, e. a., “Openimageio.”
[18] Kent, B. R., [3D Scientific Visualization with
Blender R
], 2053-2571, Morgan Claypool Publish-
ers (2015).
[19] Kent, B. R., “Visualizing astronomical data with
blender,” Publications of the Astronomical Society
of the Pacific 125(928), 731 (2013).
[20] Pham, D. L., Xu, C., and Prince, J. L., “Cur-
rent methods in medical image segmentation,” An-
nual review of biomedical engineering 2(1), 315–
337 (2000).
[21] C¸ i¸cek, ¨
O., Abdulkadir, A., Lienkamp, S. S., Brox,
T., and Ronneberger, O., “3d u-net: learning
dense volumetric segmentation from sparse anno-
tation,” in [International conference on medical
image computing and computer-assisted interven-
tion], 424–432, Springer (2016).
[22] Sahoo, P., Soltani, S., and Wong, A., “A survey
of thresholding techniques. comput vision graph
image proc 41: 233–260,” (1988).
[23] Adams, R. and Bischof, L., “Seeded region grow-
ing,” IEEE Transactions on pattern analysis and
machine intelligence 16(6), 641–647 (1994).
[24] Lorensen, W. E. and Cline, H. E., “Marching
cubes: A high resolution 3d surface construction
algorithm,” in [ACM siggraph computer graphics],
21(4), 163–169, ACM (1987).
[25] Selan, J., “Cinematic color: from your monitor
to the big screen,” in [ACM SIGGRAPH 2012
Courses], 9, ACM (2012).
[26] Rueden, C. T., Schindelin, J., Hiner, M. C., DeZo-
nia, B. E., Walter, A. E., Arena, E. T., and Eliceiri,
K. W., “Imagej2: Imagej for the next generation of
scientific image data,” BMC bioinformatics 18(1),
529 (2017).
[27] Lechner, P., [Multimedia programming using
Max/MSP and TouchDesigner], Packt Publishing
Ltd (2014).
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
This article presents ColorMoves, an interactive tool that promotes exploration of scientific data through artist-driven color methods in a unique and transformative way. We discuss the power of contrast in scientific visualization, the design of the ColorMoves tool, and the tools application in several science domains. ColorMoves is a freely available resource that can be found at SciVisColor.org.
Article
Full-text available
Background ImageJ is an image analysis program extensively used in the biological sciences and beyond. Due to its ease of use, recordable macro language, and extensible plug-in architecture, ImageJ enjoys contributions from non-programmers, amateur programmers, and professional developers alike. Enabling such a diversity of contributors has resulted in a large community that spans the biological and physical sciences. However, a rapidly growing user base, diverging plugin suites, and technical limitations have revealed a clear need for a concerted software engineering effort to support emerging imaging paradigms, to ensure the software’s ability to handle the requirements of modern science. Results We rewrote the entire ImageJ codebase, engineering a redesigned plugin mechanism intended to facilitate extensibility at every level, with the goal of creating a more powerful tool that continues to serve the existing community while addressing a wider range of scientific requirements. This next-generation ImageJ, called “ImageJ2” in places where the distinction matters, provides a host of new functionality. It separates concerns, fully decoupling the data model from the user interface. It emphasizes integration with external applications to maximize interoperability. Its robust new plugin framework allows everything from image formats, to scripting languages, to visualization to be extended by the community. The redesigned data model supports arbitrarily large, N-dimensional datasets, which are increasingly common in modern image acquisition. Despite the scope of these changes, backwards compatibility is maintained such that this new functionality can be seamlessly integrated with the classic ImageJ interface, allowing users and developers to migrate to these new methods at their own pace. Conclusions Scientific imaging benefits from open-source programs that advance new method development and deployment to a diverse audience. ImageJ has continuously evolved with this idea in mind; however, new and emerging scientific requirements have posed corresponding challenges for ImageJ’s development. The described improvements provide a framework engineered for flexibility, intended to support these requirements as well as accommodate future needs. Future efforts will focus on implementing new algorithms in this framework and expanding collaborations with other popular scientific software suites. Electronic supplementary material The online version of this article (doi:10.1186/s12859-017-1934-z) contains supplementary material, which is available to authorized users.
Conference Paper
Full-text available
Scientists from the Climate, Ocean and Sea Ice Modeling Team (COSIM) at the Los Alamos National Laboratory (LANL) are interested in gaining a deeper understanding of three primary ocean currents: the Gulf Stream, the Kuroshio Current, and the Agulhas Current & Retroflection. To address these needs, visual artist Francesca Samsel teamed up with experts from the areas of computer science, climate science, statistics, and perceptual science. By engaging an artist specializing in color, we created colormaps that provide the ability to see greater detail in these high-resolution datasets. The new colormaps applied to the POP dataset enabled scientists to see areas of interest unclear using standard colormaps. Improvements in the perceptual range of color allowed scientists to highlight structures within specific ocean currents. Work with the COSIM team members drove development of nested colormaps which provide further detail to the scientists.
Article
Full-text available
To minimize feature loss in T1- and T2-weighted MRI by merging multiple MR images acquired at different TR and TE to generate an image with increased dynamic range. High Dynamic Range (HDR) processing techniques from the field of photography were applied to a series of acquired MR images. Specifically, a method to parameterize the algorithm for MRI data was developed and tested. T1- and T2-weighted images of a number of contrast agent phantoms and a live mouse were acquired with varying TR and TE parameters. The images were computationally merged to produce HDR-MR images. All acquisitions were performed on a 7.05 T Bruker PharmaScan with a multi-echo spin echo pulse sequence. HDR-MRI delineated bright and dark features that were either saturated or indistinguishable from background in standard T1- and T2-weighted MRI. The increased dynamic range preserved intensity gradation over a larger range of T1 and T2 in phantoms and revealed more anatomical features in vivo. We have developed and tested a method to apply HDR processing to MR images. The increased dynamic range of HDR-MR images as compared to standard T1- and T2-weighted images minimizes feature loss caused by magnetization recovery or low SNR.
Article
This paper presents CHIMERA, a third-generation digital holographic printing system that solves the known problems of the two previous generations. This holoprinter is based on the use of low-power RGB continuous lasers combined with the ultrafine-grain silver-halide material Ultimate U04 and is capable of printing at a frequency equal to or greater than 25 hogels per second, full-color, 120° full-parallax digital reflection holograms or holographic optical elements with a size of up to $60 \times 80\,\,{\rm{cm}}$60×80cm and a hogel size ranging from 250 to 500 µm. A 3D scanner using a 4K video camera has been specially designed for scanning real objects printable on CHIMERA, which offers new achievements in terms of color rendition, palette, and accuracy and opens new perspectives for digital holography applications and holography in general.
Conference Paper
This paper introduces a network for volumetric segmentation that learns from sparsely annotated volumetric images. We outline two attractive use cases of this method: (1) In a semi-automated setup, the user annotates some slices in the volume to be segmented. The network learns from these sparse annotations and provides a dense 3D segmentation. (2) In a fully-automated setup, we assume that a representative, sparsely annotated training set exists. Trained on this data set, the network densely segments new volumetric images. The proposed network extends the previous u-net architecture from Ronneberger et al. by replacing all 2D operations with their 3D counterparts. The implementation performs on-the-fly elastic deformations for efficient data augmentation during training. It is trained end-to-end from scratch, i.e., no pre-trained network is required. We test the performance of the proposed method on a complex, highly variable 3D structure, the Xenopus kidney, and achieve good results for both use cases.
Article
We propose a method to adjust the luminance mapping of medical images on high-dynamic-range (HDR) display devices that minimizes perceptual and hardware veiling glare effects. We utilize the DICOM grayscale standard display function to compute the maximum number of just-noticeable-differences (JND) for an HDR prototype (dual-layer LCD). Using previous findings, a unique image will be displayed with a front and back LCD panel such that the combined light modulation produces the most accurate luminance representation of the image and the least hardware glare. Next, we use an empirical, image-dependent model to analyze regions of interest in the image that may suffer from perceptual veiling glare and adjust the luminance mapping until a reasonable degradation tolerance for detection thresholds is reached.
Article
Color affects many areas of the computer-graphics pipeline, From texture painting to lighting, rendering, compositing, image display, and the theater, handling color is a tricky problem. Tired of getting your images right on the monitor only to have them fall apart later on? This course presents the best practices used in modern visual-effects and animation-color pipelines, and how to adapt apply these concepts for home use. The course begins with an introduction to color processing and its relationship to image fidelity, color reproducibility, and physical realism. Topics include: common misconceptions about linearity, gamma, and working with high-dynamic-range (HDR) color spaces. Pipeline examples from recent films by Sony Pictures Imageworks explain which color transforms were used and why. The course concludes with a brief discussion of recent developments in color standardization at the Academy of Motion Picture Arts & Sciences and how attendees can experiment with all of these concepts for free using open-source software.