Sttreaming Video Textures for Mixed Reality Applications in Interactive Ray Tracing Environments.
ABSTRACT The realm of mixed reality applications lies in blending rendered images with images of the real world. This requires highly realistic rendered im- ages in order to seamlessly blend between those two worlds. However, current rasterization technology severely limits the achievable realism and imposes strict limits on the scene complexity and the optical effects that can be simulated efficiently. Real-time ray tracing can overcome many of these constraints and enables completely new approaches for mixed reality applications. This paper explores this design space based on a framework for live streaming of video textures in a real-time ray tracing engine. We also sug- gest a novel approach to video-based AR by inte- grating image compositing with shading computa- tions. We demonstrate the approach with a num- ber of VR/AR applications including video inserts, video billboards, and dynamic lighting from video and HDR video streams. Being seamlessly inte- grated into the ray tracing framework, all our ap- plications feature ray traced effects, like shadows, reflections and refraction.
- SourceAvailable from: psu.edu[Show abstract] [Hide abstract]
ABSTRACT: In the last years, the well known ray tracing algorithm gained new popularity with the introduction of interactive ray tracing methods. The high modularity and the ability to produce highly realistic images make ray tracing an attractive alternative to raster graphics hardware.Interactive ray tracing also proved its potential in the field of Mixed Reality rendering and provides novel methods for seamless integration of real and virtual content. Actor insertion methods, a subdomain of Mixed Reality and closely related to virtual television studio techniques, can use ray tracing for achieving high output quality in conjunction with appropriate visual cues like shadows and reflections at interactive frame rates.In this paper, we show how interactive ray tracing techniques can provide new ways of implementing future virtual studio applications.Journal of Virtual Reality and Broadcasting. 01/2005;
Conference Paper: Virtual jewel rendering for augmented reality environments[Show abstract] [Hide abstract]
ABSTRACT: Virtual Mirrors are augmented reality applications that capture a viewer's image and add 2D or 3D overlaid virtual elements, such as cloth patterns, or replace existing pieces of garments, such as shoes, by virtual ones. The purpose of our study is to focus on the addition of jewels that present complex rendering issues due to reflection and translucency, especially for real-time applications. We propose two complementary approaches which are both suited for particular objects. The image-based rendering technique relies on an image data-set of photos taken on a semi-circle of camera positions and a 3D reconstruction of the jewel. This approach is exploited to demonstrate real jewels with sophisticated refractions and highlights. The analytical 3D rendering technique can also visualize configurable virtual jewels and is based on the decomposition of the jewel into parts without non-local reflection or transparency. A graph-based rendering chain takes in consideration some of the self-reflection and lighting effects.Image Processing (ICIP), 2010 17th IEEE International Conference on; 10/2010
- [Show abstract] [Hide abstract]
ABSTRACT: This paper introduces new approaches to enable collaborative outdoor mixed reality design review in the architectural domain as well as outdoor mixed reality experiences in the cultural heritage domain. For this purpose we present the results of three closely related European projects, IMPROVE and CINeSPACE, which are currently succeeded by MAXIMUS, continuing the development of technologies relevant to the two domains. The paper focuses on the base technologies needed to develop usable outdoor mixed reality applications, such as marker-less optical tracking combined with sensor fusion for accurate pose estimation outdoors. Furthermore the paper presents a new visualization system developed within the project, one of the few daylight blocking head.-mounted displays available. In addition to pose estimation and visualization devices, the paper presents a VR framework adapted for the architecture and the cultural heritage domain featuring high dynamic range image acquisition combined with pre-computed radiance transfer rendering to be able to photo-realistically render urban content. For the cultural heritage domain this framework has been extended to allow accurate display and generation of multimedia content super-imposed on a real environment as well as city navigation.Proceedings of the 2010 Spring Simulation Multiconference, SpringSim 2010, Orlando, Florida, USA, April 11-15, 2010; 01/2010
Streaming Video Textures for Mixed Reality Applications in
Interactive Ray Tracing Environments
Andreas Pomi, Gerd Marmitt, Ingo Wald, and Philipp Slusallek
Saarland University, Computer Graphics Group
Im Stadtwald - Building 36.1, 66123 Saarbr¨ ucken, Germany
Figure 1: Some example mixed reality applications using our framework: a) A virtual room illumi-
nated by live video on the TV set, b) Two video billboards integrated into a scene with correct shadows
and reflections, and c) A virtual car in a real environment, also with ray traced shadows and reflections.
The realm of mixed reality applications lies in
blending rendered images with images of the real
world. This requires highly realistic rendered im-
ages inorder toseamlessly blend between those two
worlds. However, current rasterization technology
severely limits the achievable realism and imposes
strict limits on the scene complexity and the optical
effects that can be simulated efficiently. Real-time
ray tracing can overcome many of these constraints
and enables completely new approaches for mixed
This paper explores this design space based on
a framework for live streaming of video textures
in a real-time ray tracing engine.
gest a novel approach to video-based AR by inte-
grating image compositing with shading computa-
tions. We demonstrate the approach with a num-
ber of VR/AR applications including video inserts,
video billboards, and dynamic lighting from video
and HDR video streams. Being seamlessly inte-
grated into the ray tracing framework, all our ap-
plications feature ray traced effects, like shadows,
reflections and refraction.
We also sug-
Mixed reality applications range from augment-
ing videos with photo-realistic synthetic objects
(e.g. in design review applications or assembly
guides) up to inserting real persons into virtual
worlds (e.g. virtual TV studios  or immersive
tele-conferencing). For combining the real and the
synthetic world the renderer needs as much infor-
mation about the real world as possible, including
accurate geometry, lighting conditions, and material
properties. Due to its ability to correctly simulate
light transport, the ray tracing algorithm is perfectly
suited for this task. However, it has been much too
slow in the past.
Recent advances in real-time ray tracing sys-
tems  provide new opportunities for high-
quality rendering including physically correct light-
ing, reflection, and refraction effects, which paves
the way to novel interactive applications. This sug-
gest that it is well worth exploring this design space
for interactive mixed reality applications.
Since the real-time reconstruction of accurate 3D
models of non-trivial real environments is still very
limited, interactive applications often use rather
simple models, such as live video captured from the
surrounding environment. The integration of live
video into the ray tracing process is therefore a nec-
VMV 2003Munich, Germany, November 19–21, 2003
essary first step for realizing AR/VR applications.
In the following we provide a framework for
streaming and synchronizing live video in a dis-
tributed ray tracing system using it as video textures
in the shading process. We also suggest a method
for compositing video streams and normal shading
data for video-based augmented reality (AR) ap-
plications . Note that the term video textures
is also used for pseudo-randomly looped, animated
textures introduced by Sch¨ odel et al. . In the
remainder of this paper we use the term to refer to
textures from live video streams.
The rest of this paper is structured as follows: In the
next section we briefly review real-time ray tracing
and discuss the problems arising in the context of
integrating video textures into a distributed ray trac-
ing engine. We suggest a framework for integrating
streamingvideo textures intherendering system us-
ing multicast networking (Section 3) and stream-
ing AR view information (Section 4.1). We then
describe our implementation based on the OpenRT
system (Section 4) and discuss a number of sample
applications that benefit fromour modular approach
(Section 5). Finally we summarize and discuss our
resultsand suggest future applications andimprove-
2 Video Textures for Distributed Ray
Recently real-time ray tracing has become feasible
due to algorithmic improvements, optimized imple-
mentations, and distributed computing on a cluster
of commodity PCs . Already today, real-time
ray tracing enables novel applications such as inter-
active global illumination  or the visualization of
complex refraction and reflection scenarios . A
new API for ray tracing (OpenRT ) that is simi-
lar to OpenGL simplifies the design and implemen-
tation of new applications profiting from the ad-
vanced optical effects and the high modularity of
While hardware support for ray tracing is being
developed [20, 19, 17], these technologies are not
yet available. High rendering quality at realisticres-
olutions and frame rates still requires to use parallel
and distributed ray tracing, for example by using a
cluster of PCs.
A typical distributed ray tracing architecture con-
sists of a rendering server and a number of render-
ing clients connected by commodity networks (see
Figure 2a). The server runs the ray tracing appli-
cation which communicates with the rendering en-
gine through the OpenRT API . The ray tracing
library transparently performs the distribution and
parallelization of rendering jobs across the clients
by splitting the frame buffer into rectangular tiles
on request. As long as enough jobs are available
this allows the rendering performance to scale effi-
ciently by simply adding more rendering clients.
For video textures we need a way to stream the
video information to all rendering clients.
simplest way – an explicit network connection
(TCP/IP) to each client – does not scale with the
number of clients as the network bandwidth of the
server increases linearly with each new client.
As an alternative we could use a demand driven
approach, having clients requesting texture data
from a video server on demand. Because not all
clients need all texels, this would reduce the net-
work bandwidth. However, the high latency due
to the commodity network is not tolerable for real-
time applications. Another alternative would be to
provide the video information via a separate wiring
to the clients, and installing a video frame grabber
ineach client. However, thisapproach istechnically
not very practical and would require even more so-
phisticated synchronization methods.
Because all clients might require some part of
the video data, a 1-to-N communication mechanism
such as IP multicast seems the best solution. This
is particularly true for local networks that minimize
the routing issues of multicast traffic. However, it
requires that the system can cope with synchroniza-
tion issues and packet losses typically appearing in
????????? pixels) and distributing them to clients
3Video Streaming with IP Multicast
IP multicast transport  has been designed for
scalable streaming of audio and video across the In-
ternet. It provides a simple solution for distributing
data to a number of hosts in parallel. The multicast
IP address represents not a single host but a group
of hosts and multiple hosts can send packets on the
Figure 2: a) OpenRT distributed ray tracing system. Each client (C) has a (reliable) TCP/IP connection to
the rendering server (S). The server has a higher bandwidth network connection (e.g. Gigabit). b) System
extended with streaming video textures. The video texture server (V) sends sync information and texture
data on separate (unreliable) multicast groups.
same group. Hosts can dynamically join and leave
multicast groups. The network routes the packets
to hosts that have subscribed to a stream and copies
the packets at routers and switches as required.
Due to the 1-to-N relation between sender and
the receivers IP multicast only works for UDP data-
gram packets. The lack of an acknowledging mech-
anism causes reliability issues with traditional mul-
ticast, i.e. there is no way to guarantee that packets
have been properly delivered.
Loss of packets in LAN environments is mostly
caused by packet queue overflows within switches
and the operating system.
ful control of the maximum network bandwidth.
Several protocols (e.g. Reliable Multicast Protocol
RMP ) have been designed to increase the reli-
ability of IP multicast. However, they are designed
for WAN use and increase latency, which makes
them unsuitable for interactive applications.
This requires care-
3.1 System Architecture
Our framework uses a video server for streaming
the video textures to the ray tracing clients (Fig-
ure 2b). The video servers run on separate hosts
to minimize the bandwidth impact on the rendering
server. A typical server uses a frame grabber board
for live video input where resolution and frame rate
can be adjusted dynamically. For testing purposes
it can also use pre-recorded video from files.
UDP packets cannot exceed a size of 64Kb on
most systems and since media like Fast Ethernet
uses packet sizes of 1500 byte, video frames need
to be split into packets before streaming. A packet
consists of a header and part of the texture color
data (e.g. one image row). The header contains in-
formation about the texture format (width, height,
color coding), the position of the texture color data
intheframe, aswell asasynchronization timestamp
(video frame number).
Distributed ray tracing systems render tilesof the fi-
nal image frame asynchronously, i.e. different hosts
might start computation of a frame at different
times. In order to avoid tiling artifacts, we need a
synchronization mechanism to assurethat all clients
rendering the tiles of one rendering frame use the
corresponding video texture frame. While the frame
rate of the video texture is fairly constant, the ren-
dering frame rate might vary depending on the cost
of the current view.
To avoid synchronization overhead, the clients
are usually unaware of the processing of all other
clients. To synchronize the clients’ video textures,
the server therefore has to perform synchroniza-
tion . If the server also listens to the streamed
video it knows which texture frames have just been
transmitted to the clients and can distribute this in-
formation with each new frame.
Because therendering server isalready anetwork
bottleneck we should not increase its load with the
video data. As it only needs timestamps and not the
video content, we transmit small packets containing
the texture ID and the timestamp of the last frame
sent on a separate multicast group. Thus the server
must only receive this sync information for making
its global decision. Once the server has decided on
the video texture frame to be used for the given ren-
dering frame, it can safely sent this single integer
value to all clients via reliable unicast without hav-
ing to consider bandwidth limitations.
3.3 Data Formats
Video texture data comprises not only the color in-
formation for the texels but also the resolution of
a texture frame (width and height) and a format
identifier. The latter is used for decoding and cor-
rect texture interpolation. A texture interpolator is
a function to map texture coordinates to memory
addresses with additional texture filtering (e.g. bi-
By providing a table of those interpolators we
can use different texture coding formats, from
packed 16bit RGB (RGB565) up to formats con-
taining an alpha channel or even special formats
like RGBE  for videos with high dynamic range
data. One could also use data compression to de-
crease the bandwidth or exploit scalable video cod-
ing with multi-resolution formats for hiding packet
losses (e.g. MPEG).
In addition, we sometimes require streaming of
arbitrary data structures (e.g. a table of light sam-
ples, see Section 5.4). A raw format without inter-
polation can provide such a mechanism. It takes
special care of packet losses (e.g. by using frag-
menting writes ).
Since multicast is based on UDP packets there
is no guarantee that video packets arrive on every
client or even leave the sending host. Optimizing
buffer sizes and a carefully timed sender decrease
the amount of lost packets but still provide no reli-
ability. Instead we favor a rather simple approach:
Lostpartsof atextureframearetoleratedby reusing
the information stored in previous frames. In prac-
tice this seems to work well if single rows get lost.
’Burst losses’ can easily be hidden by sending the
image rows in random order.
4 Implementation Within OpenRT
A major goal of our implementation in the OpenRT
system  was to leave the rendering kernel un-
changed. OpenRT is expandable by a number
of run-time loadable plug-ins like shaders, camera
models, light source types etc. Thismakes a plug-in
implementation of streaming video textures simple
The plug-in on the server receives the informa-
tion from the sync multicast group, updates a ta-
ble listing the latest timestamp for each video tex-
ture, and forwards it to clients for new frames. A
Video Texture Manager on each client provides ac-
cess to the textures for any shader. It creates a sep-
arate thread for receiving the data and manages the
synchronization. A separate texture object for each
video stream allows multiple shaders to access the
same video. Via unique IDs assigned by the video
server the application can dynamically switch be-
tween multiple video textures used by a specific
A video texture server is a stand-alone applica-
tion running on an arbitrary host. It uses a cus-
tom server library that provides a simple API for
streaming textures and data structures.
brary hides the synchronization mechanism to the
user. We implemented a simple server using the
Video4Linux  API for access to frame grabber
boards. Another implementation offers playback of
previously recorded video files.
By combining the video texture server library
with the Network Multimedia System NMM 
more complex scenarios could be realized. Exam-
ples could be the playback of DVDs, streaming of
digital television programs, and video conference
4.1 AR View Compositing
AR applications augment a view of the real world
with computer generated images. One option for
compositing are special AR hardware devices like
semi-transparent video glasses that physically per-
forms the composition. As an alternative, we can
use a camera to capture the user’s view and per-
form compositing in the computer (video-based
augmented reality ). In this case, the renderer
provides not only an image of the synthetic part of
the scene but also a matte in form of an alpha chan-
nel that masks the foreground objects. The matte
is used for compositing the two video images on
the rendering host  or with external keyer hard-
Obviously, one could use the same mechanisms
with the OpenRT architecture. However, we prefer
to use the built-in shaders to do the compositing op-
erations. We provide the video background color as
an input for the shading of a pixel. Hence we can
overwrite the color for opaque objects or just mod-
Figure 3: Room with TV set. a) Video texture on the TV screen with ambient lighting. b) The same scene
with the video texture acting as a light source. The TV screen is subdivided into 5x5 light samples that
illuminate the scene. The ambient term adapts to the sample average. Note the reflections on the table and
the soft shadows cast by the table’s legs.
ify it for transparent objects. This also allows us to
perform differential rendering [6, 18] for creating
lighting effects like shadows or caustics on the real
As an example we can add a shadow to a syn-
thetic object by approximating the geometry and re-
flectance of the real scene with simplified synthetic
stand-in objects (like a ’shadow catching’ plane on
the floor ). We use the shader of this plane to
modulate the background video input color accord-
ingto the ratioof incident light withand without the
synthetic objects (see application example in Sec-
tion 5.4) .
We could use the above multicast mechanism
to also stream the background video to all clients.
However, in order to minimize latency and improve
reliability the rendering server distributes this video
to the clients together with the tiles using the reli-
able TCP/IPconnection. In contrast to the multicast
method a client only receives the background for its
part of the final rendered image and not the whole
background video image. The use of TCP/IP elimi-
nates disturbing artifacts caused by packet losses.
Note that this method does not significantly in-
crease the network load compared to using the mul-
ticast approach as we are just using the unloaded
direction of the full-duplex Ethernet connection.
By using a simple shader that just returns the
background color for stand-in objects we can se-
lectively hide parts of the synthetic scene and sim-
ulate view dependent occlusion. However, as we
only have access to the background color directly
behind a shaded object this approach does not al-
low for computing refraction effects.
The above framework now allows for novel ap-
proaches by using the seamless integration of video
textures into a real-time ray tracing system. In the
following we will briefly discuss some example ap-
5.1 Live Video Inserts
The most obvious application of streaming video
textures is live video inserting, i.e. using the video
stream as a dynamically changing texture for any
type of shader.
The scene shown in Figure 3a consists of a sim-
ple living room scenario with a television set. The
shader used for the screen material simply returns
the color from the appropriate video texel. The inte-
gration into a ray tracing system automatically pro-
vides the reflection on the table and other reflective
5.2 Dynamic Lighting from Video Tex-
Figure 3b shows the same scene but this time the
room is illuminated by the TV set. To achieve this
effect we simply placed an array of point lights in
front of the TV screen. The light source shaders
have access to the video texture, and compute their
color and intensity based on the texture data. We
also use the video texture to control the level of am-
bient light in the room. Since lighting is calculated
from scratch for each frame the direct illumination
in the scene resembles the light emitted by the TV
Figure 4: Video billboards: Chroma keying is done in the shader, which automatically makes the keying
information available for other shaders computing shadows, reflections and refractions. a) Two persons in
a virtual environment. The persons are captured separately in front of a green-screen. b) The billboard
objects without video textures. c) Refraction of a person standing behind a glass sphere.
This scenario can easily be extended for indirect
lighting by using the instant global illumination al-
gorithm of the OpenRT system [24, 3].
5.3 Video Billboards
Another interesting application of video textures
is augmenting a rendered scene with real persons
e.g. for tele-presence applications or virtual TV
sets. The input video from a camera is segmented
into foreground and background using background
subtraction or chroma keying methods . How-
ever, instead of segmenting the signal on the video
server and streaming the signal together with the
matte information (alpha channel) we can take ad-
vantage of distributed ray tracing and perform the
segmentation also in the shader. This reduces the
bandwidth and performs segmentation only for tex-
els actually visible in the current frame.
The areas of a video billboard segmented as fore-
ground simply show the video texture, possibly
modified by the shader, while the background area
is computed by tracing a transparency ray. Since
this works not only for primary but for arbitrary
rays, shadow and reflection effects even on curved
surfaces are simulated without any additional effort
(Figures 4 and 7).
Due to their 2D nature billboards have some
drawbacks: They can look distorted for shallow
viewing angles. Furthermore, if the person moves
back and forth relative to the video camera, the sil-
houette may move up and down in the video image
causing the person to ’hover’ in the composite im-
age. A 3D reconstruction of the person using a vi-
sual hull  and a volume shader combined with
video textures for the view cameras would provide
a more realistic solution. However, this has not yet
5.4Dynamic Lighting With Real World
Video textures and real-time ray tracing also allow
for interactive lighting of virtual objects with real-
time captured lighting. Together with a live back-
ground video stream (see Section 4.1) this signifi-
cantly improves the realism of AR scene (Figures 5
We use a video camera with remotely control-
lable shutter time together with a 180 degree fish-
eye lens to capture the incident light in the upper
hemisphere (Figure 5a). By quickly changing the
shutter times and capturing frames at different ex-
posure levels we can reconstruct a high dynamic
range (HDR) video stream [7, 14]. HDR calcula-
tion is done on a special video texture server that
then streams a high dynamic range video in RGBE
format . This video texture server also samples
the HDR lightprobe image and streams a table of
directional light samples to all clients.
The RGBE video texture is then applied as an en-
vironment map  for illumination  and for
specular reflections. A plane under the object acts
as stand-in for the real floor and is used for shadow
generation [6, 18].
Most of the applications shown here could also be
implemented using rasterization hardware. How-
ever, they are much easier and straightforward to
180 degree fish−eye
AR view camera
Figure 5: Dynamic lighting of a virtual car. a) Setup: A video camera captures the scene view while a
second camera with a 180 degree fish-eye lens is used to capture the incident light in the upper hemisphere.
b) The background with the lightprobe. c) The final composited image. Note the soft shadow on the floor.
TV w/o lighting
TV w. lighting
BBD w. lighting
BBD w. glass sphere
3a + 6
3b + 6
4a + 7
4c + 7
5 + 8
130x100 @ 25fps
130x100 @ 25fps
220x420 @ 25fps
220x420 @ 25fps
350x350 @ 1fps
Table 1: Frame rates achieved on our example applications. Note that due to limitations in camera speed
the environment map in the HDR lighting example (car) is updated at only 1 fps.
implement in a ray tracing environment. The mod-
ularity of an OpenRT based rendering system as-
sures high re-usability of code like shaders and easy
combination of rendering effects likereflections, re-
fraction, and shadows. All shading effects combine
automatically and correctly with all other objects in
The frame rates in the example applications are
given in Table 1. The rendering clients are dual
Athlon MP 1800+ PCs. They are connected via
FastEthernet (100Mbit/s) to a multicast capable
switch. The rendering server is connected via a Gi-
gabit (1000Mbit/s) uplink. No compression is used
for video textures.
The AR view composition method presented
in Section 4.1 runs at 19 fps at video resolution
(640x480). For lower resolutions the full video
frame rate of 25 fps is being achieved. Note that
our current OpenRT implementation is limitedto19
fps for resolutions higher than 640x480 due to load
balancing issues. Video delay is about four frames.
No change in the rendering frame rate is noticeable
when switching video streaming on and off in the
AR view compositing.
In this paper we presented a framework that extends
the real-time OpenRT system by using streaming
video textures and multicast networking. We have
demonstrated its use with several AR and mixed re-
Compared to sophisticated rasterization hard-
ware with video stream support used for hard real-
time applications (i.e. virtual TV studios), inter-
active ray tracing on commodity hardware cannot
provide the low latency in terms of video delay in
combination with fixed frame rates (e.g. PAL 50
fields rendering in TV applications) yet. This is
mainly caused by distribution aspects like low net-
work bandwidth and load balancing issues. For ap-
plications with weaker requirements where image
quality and modularity in terms of programming is
important, interactiveraytracingprovides acheaper
and more flexible alternative compared to special-
7.1 Future Work
The streaming mechanism described here could
be improved by frame rate feedback to the video
servers. The existing synchronization multicast
group could be used for this purpose. The video
server could then adapt sending rate of the video to
the rendering frame rate.
Furthermore, the video texture bandwidth could
easily be reduced by introducing image compres-
sion. Compression and error-correction schemes
should be used that can correct or hide lost pack-
In the future we will concentrate our work on the
3D reconstruction briefly mentioned in Section 5.3.
This would allow for real 3D compositing and bet-
ter integration of the real and virtual worlds. We
believe that 3D compositing is a major key for real-
istic mixed reality rendering.
The authors wish to thank J¨ org Schmittler for the
TV room scene and also Andreas Dietrich, Carsten
Benthin, Kim Herzig, Simon Hoffmann, and Chris-
tian Linz for their help.
This work has been supported by Intel Corp.
 R.Azuma. A survey ofaugmented reality. Computer Graph-
ics (SIGGRAPH’95 Proceedings, Course Notes #9: Devel-
oping Advanced Virtual Reality Applications), pages pp. 1–
 Carsten Benthin, Ingo Wald, Tim Dahmen, and Philipp
Slusallek. Interactive Headlight Simulation – A Case Study
of Distributed Interactive Ray Tracing. In Proceedings of
Eurographics Workshop on Parallel Graphics and Visualiza-
tion (PGV), pages 81–88, 2002.
 Carsten Benthin, Ingo Wald, and Philipp Slusallek. A Scal-
able Approach to Interactive Global Illumination. to be pub-
lished at Eurographics 2003, 2003.
 Ron Brinkmann. The Art and Science of Digital Composit-
ing. Morgan Kaufmann Publishers, Inc., first edition, 1999.
 Chris Buehler, Wojciech Matusik, Leonard McMillan, and
Steven Gortler. Creating and rendering image-based visual
hulls. Technical Report MIT/LCS/TR-780, Massachusetts
Institute of Technology, 1999.
 Paul Debevec. Rendering synthetic objects into real scenes:
Bridging traditional andimage-based graphicswith globalil-
lumination and high dynamic range photography. Computer
Graphics, 32(Annual Conference Series):189–198, 1998.
 Paul E. Debevec and Jitendra Malik. Recovering high dy-
namic range radiance maps from photographs. Computer
Graphics, 31(Annual Conference Series):369–378, 1997.
 Andreas Dietrich, Ingo Wald, Carsten Benthin, and Philipp
Slusallek. The OpenRT Application Programming Interface
– Towards A Common API for Interactive Ray Tracing. In
Proceedings of the 2003 OpenSG Symposium, Darmstadt,
Germany, 2003. Available at http://www.openrt.de.
 NMM Network-Integrated Multimedia Middleware for
 Oliver Grau and Graham Thomas.
3D modelling techniques in broadcast applications. 2002
Tyrrhenian International Workshop on Digital Communica-
 Ned Greene. Environment mapping and other applications
of world projections. IEEE CG&A, 6(11):21–29, November
 Jack Keith. Video Demystified. A Handbook for the Digital
Engineer. Harris Semiconductor, second edition, 1996.
 Doug Kelly. Digital Compositing In Depth. Coriolis, 2000.
 Steve Mann and Rosalind W. Picard. Being ’undigital’ with
digital cameras: Extendingdynamic range bycombining dif-
ferently exposed pictures, 1995.
 Gene S. Miller and C. Robert Hoffman. Illumination and
reflection maps: Simulated objects in simulated and real en-
vironments. SIGGRAPH 84 Advanced Computer Graphics
Animation seminar notes, 1984.
 Video4Linux Project. http://www.exploits.org/v4l/.
 Timothy J.Purcell, Ian Buck, William R.Mark,and Pat Han-
rahan. Ray Tracing on Programmable Graphics Hardware.
ACM Transactions on Graphics, 21(3):703–712, 2002. (Pro-
ceedings of SIGGRAPH 2002).
 Imari Sato, Yoichi Sato, and Katsushi Ikeuchi. Acquiring
a radiance distribution to superimpose virtual objects onto a
real scene. IEEE Transactions on Visualization and Com-
puter Graphics, 5(1):1–12, 1999.
 J¨ org Schmittler, Alexander Leidinger, and Philipp Slusallek.
A Virtual Memory Architecture for Real-Time Ray Tracing
Hardware. to appear in Computer and Graphics, Volume 27,
No. 5, 2003.
 J¨ org Schmittler, Ingo Wald, and Philipp Slusallek. SaarCOR
– A Hardware Architecture for Ray Tracing. In Proceedings
of Eurographics Workshop on Graphics Hardware, pages
 Arno Sch¨ odl, Richard Szeliski, David H. Salesin, and Ir-
fan Essa.Video textures.
graph 2000, Computer Graphics Proceedings, pages 489–
498. ACM Press / ACM SIGGRAPH / Addison Wesley
 Alvy Ray Smith and James F. Blinn. Blue screen matting.
In Proceedings of the 23rd annual conference on Computer
graphics and interactive techniques, pages 259–268. ACM
 W.Richard Stevens. UNIX NetworkProgramming.Network-
ing APIs: Sockets and XTI. Prentice-Hall, second edition,
 Ingo Wald, Thomas Kollig, Carsten Benthin, Alexander
Keller, and Philipp Slusallek. Interactive Global Illumina-
tion using Fast Ray Tracing. Rendering Techniques 2002,
pages 15–24, 2002. (Proceedings of the 13th Eurographics
Workshop on Rendering).
 Ingo Wald and Philipp Slusallek. State-of-the-Art in Interac-
tive Ray-Tracing. In State of the Art Reports, Eurographics
2001, pages 21–42, 2001.
 Greg Ward. Real pixels. In James Arvo, editor, Graphics
Gems II. Academic Press, 1992.
 Brian Whetten.A reliable multicast protocol.
and Practice in Distributed Systems. Birman, Mattern and
Schiper, eds., Lecture Notes on Computer Science, 938,
Use of image-based
In Kurt Akeley, editor, Sig-
Figure 6: Room with TV set. The video texture is used for both: providing the video image on the TV
screen and controlling the direct lighting of the room. Note the shadows on the floor and the reflections on
the table. Left top, left bottom and center top: Different lighting situations. Center bottom: Without lighting
by the TV set.
Figure 7: Video billboards. Left top: With and without shadow and reflection. The person seems to hover
without shadow. Left bottom: Glass refraction and reflections. Right: Two persons rendered with separate
Figure 8: Dynamic AR lighting. Left: Background video for AR view compositing and fish-eye HDR
lightprobe. Right: Car model composed over background. Note the soft shadow on the real floor.