ArticlePDF AvailableLiterature Review

Augmented Reality and Virtual Reality Displays: Perspectives and Challenges


Abstract and Figures

As one of the most promising candidates for next-generation mobile platform, augmented reality (AR) and virtual reality (VR) have potential to revolutionize the ways we perceive and interact with various digital information. In the meantime, recent advances in display and optical technologies, together with the rapidly developing digital processers, offer new development directions to advancing the near-eye display systems further. In this perspective paper, we start by analyzing the optical requirements in near-eye displays poised by the human visual system and then compare it against the specifications of state-of-the-art devices, which reasonably shows the main challenges in near-eye displays at present stage. Afterward, potential solutions to address these challenges in both AR and VR displays are presented case by case, including the most recent optical research and development that are already or have the potential to be industrialized for extended reality displays.
Content may be subject to copyright.
Journal Pre-proof
Augmented reality and virtual reality displays: Perspectives and challenges
Tao Zhan, Kun Yin, Jianghao Xiong, Ziqian He, Shin-Tson Wu
PII: S2589-0042(20)30585-X
Reference: ISCI 101397
To appear in: ISCIENCE
Please cite this article as: Zhan, T., Yin, K., Xiong, J., He, Z., Wu, S.-T., Augmented reality and
virtual reality displays: Perspectives and challenges, ISCIENCE (2020), doi:
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition
of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of
record. This version will undergo additional copyediting, typesetting and review before it is published
in its final form, but we are providing this version to give early visibility of the article. Please note that,
during the production process, errors may be discovered which could affect the content, and all legal
disclaimers that apply to the journal pertain.
© 2020 The Author(s).
Augmented reality and virtual reality displays:
Perspectives and challenges
Tao Zhan,
Kun Yin,
Jianghao Xiong,
Ziqian He,
and Shin-Tson Wu
College of Optics and Photonics, University of Central Florida, Orlando, FL 32816, USA
These authors contributed equally
As one of the most promising candidates for next-generation mobile platform, augmented reality (AR)
and virtual reality (VR) have potential to revolutionize the ways we perceive and interact with various
digital information. In the meantime, recent advances in display and optical technologies, together with
the rapidly developing digital processers, offer new development directions to advancing the near-eye
display systems further. In this perspective paper, we start by analyzing the optical requirements in near-
eye displays poised by the human visual system and then compare it against the specifications of state-of-
the-art devices, which reasonably shows the main challenges in near-eye displays at present stage.
Afterward, potential solutions to address these challenges in both AR and VR displays are presented case
by case, including the most recent optical research and development that are already or have the potential
to be industrialized for extended reality displays.
As the most critical information acquisition medium, information displays have been developing rapidly
after the third industrial revolution. From the beginning of this millennium, display technologies have
successfully evolved from the bulky cathode ray tube to compact flat panel designs, such as liquid crystal
display (LCD) and organic light-emitting diode (OLED) (Chen et al., 2018). More recently, the next-
generation display technologies under dedicated development are no longer limited to flat panels that just
placed in front of the users but aimed at revolutionizing the way of interactions between the users and
their surrounding environment (Cakmakci et al., 2006). At one end of the spectrum is virtual reality (VR)
display, which effectively extends the field of view (FOV), blocks the entire ambient, and offers an
immersive virtual environment independent of the user’s real surroundings. At the other end of the
spectrum is augmented reality (AR) display, which not only pursues high-quality see-through
performance but also enriches the real world by overlaying digital contents. With advanced level of
optical technology and refreshing user experience, AR and VR displays exhibit potential to trigger
attractive applications, including but not limited to healthcare, education, engineering design,
manufacturing, retail, and entertainment.
The ideal goal of AR and VR display development is to offer reality-like crystal-clear images that can
simulate, merge into, or rebuild the surrounding environment and avoid wearing discomfort concurrently.
This is still challenging at the present stage, especially for AR systems, since most components demand
not only further performance enhancement but also miniaturization in both form factor and power
In this paper, we share a few perspectives about the development of optical technologies for AR and VR
head-mounted displays. We begin the discussion by reviewing the visual requirement poised by the
human visual systems. Next, we discuss how emerging optical technologies can help meet these
challenges in terms of resolution, visual comfort, FOV, and dynamic range. Moreover, form factor and
power efficiency are also taken into consideration because they play crucial roles in near-eye display
designs, especially for consumer applications.
To better understand the goal and underlying challenges, it is necessary to examine the performance
parameters of human visual system. The FOV has the distribution plotted in Figure 1A. The monocular
FOV of human eye is about 160° (horizontal) by 130° (vertical). The combined binocular FOV is about
200° (horizontal) by 130° (vertical), with an overlapped region of 120° horizontally (Wheelwright et al.,
2018). The resolution limit of human eye is determined by the average spacing of cone cells in the fovea.
This estimation yields the visual angle of about 0.5 arcmins (Curcio et al., 1990), or 120 pixel-per-degree
(ppd), which corresponds to 20/10 visual acuity. As it comes to display design, there is an apparent trade-
off between resolution density and FOV, given that the total number of display pixels is fixed.
For VR, a broad FOV that covers the human visual range is relatively easy to achieve by designing an
eyepiece with sufficiently low f/#. The main issue becomes the resultant low-resolution density, which
brings up the so-called screen-door effect that considerably compromises the viewing experience. A direct
solution, of course, is to increase the display resolution, which is unfortunately very challenging
considering the high cost and data transport rate. For estimation, to achieve a monocular vision with 100°
FOV and resolution density of 60 ppd (1 arcmin, or 20/20 vision), a display with 6K resolution in
horizontal is required. Some commercial products (like Pimax Vision 8K) now can provide about 4K
monocular resolution, but the daunting price that comes with the high performance remains an issue.
Another approach considers the fact that the high-resolution density only exists within the fovea region of
±2.5° (Rossi and Roorda, 2010), out of which the visual acuity drops drastically (Figure 1B). Therefore,
the high resolution is only required in the central viewing zone, which brings out the concept of foveated
display (Tan et al., 2018; Kim et al., 2019). In foveated displays, the resolution is variant across the entire
viewing region, usually through an optical combination of two display panels that individually address
central and peripheral areas. This way, not only the burden of display hardware is lessened, the
computational and data-transferring burdens are also reduced significantly.
Regarding AR systems, although the trade-off between FOV and resolution density still exists, a more
significant concern is to produce a decent FOV in the first place. Throughout various optical architectures
from free-space combiners, total internal reflection (TIR) freeform combiners (Hua, et al., 2013) to
lightguide combiners, the maximum achievable FOV typically does not exceed 60° in horizontal, which
still has a long path to go towards the human vision limit.
Furthermore, as a high-dynamic-range imaging system, the human eye can adapt to a broad range of
illuminance from 10
lux of daylight to 10
lux at night (Hoefflinger, 2007). Thus, contrast ratio (CR) is a
critical display parameter. In VR, the issue of contrast is not significant because the influence of
environment light can be neglected. If the stray light inside the headset can be well managed and
suppressed, then CR can reach over 1000:1. In AR, however, due to the high surrounding illuminance, the
requirement for display brightness can be very high. In this case, a more representative parameter to
consider is ambient contrast ratio (ACR), defined as (Lee et al., 2019):
on ambient
off ambient
+ ⋅
+ ⋅
where L
) represents the display luminance of on- (off-) state, and T is display transmittance. For a
simple estimation, if we assume a display transmittance of 80% and ambient illuminance of 10
lux with
Lambertian distribution, an ACR of 2:1 that barely prevents image washout already requires 2500 nits of
display brightness. A better CR of 5:1 for adequate readability even requires 10,000 nits of brightness.
Current AR systems, for comparison, generally can support brightness only up to 500 nits (Lee et al.,
2019), which can only accommodate indoor use (500 lux).
When evaluating the VR/AR systems capable of 3D image generation, yet another aspect to consider for
human vision is the stereo sensation. The natural viewing experience of a 3D object induces vergence cue
(relative rotation of eyes) and accommodation cue (the focus of eyes), which coincide with each other
(Figure 1C). However, in most of current VR systems, a fixed display plane with different rendered
contents for each eye is adopted. The eye accommodation is fixed on the plane and therefore mismatches
with vergence cue, which causes visual fatigue and discomfort, sabotages stereo acuity, and distorts
perceived depth (Hoffman et al., 2008, Watt et al., 2005). This phenomenon is often called vergence-
accommodation conflict (VAC).
The current angular resolution of VR displays still falls short of normal 20/20 vision acuity. Most VR
headsets are using one display panel and viewing optics for each eye to provide the stereoscopy effect;
such an old technology can trace back to the nineteenth century (Wheatstone, 1838). The VR optical
layout is essentially an unsophisticated imaging system using the viewing optics to magnify the display
panel. Therefore, from the system perspective, clearer and sharper imagery can be offered by further
improving both display panels and magnifying lenses. The display industry has been pursuing display
panels with higher resolution, power efficiency, dynamic range, and faster response time yet lower cost.
The fast-evolving flat panel display in the past decade is one of the cornerstones of current VR headsets,
and their future development will also considerably benefit the VR industry. It is vital to increase the
pixel number and density on physical display panels and thus reduce the screen-door effect in the long
term. However, this may bring a heavy burden on image rendering, driving circuits, and power
In the meantime, some emerging approaches can offer decent visual experience based on the off-the-shelf
display panels (Figure 2). For global resolution enhancement, the conventional wobulation method (Allen
and Ulichney, 2005) designed for projection displays can be extended to VR. Lee et al., 2017
demonstrated an optical wobulation VR system by synchronizing a switchable liquid crystal
Pancharatnam-Berry phase deflector and subframe images, increasing the pixel density through time-
multiplexing. Zhan et al., 2019 further advanced this approach using a passive polymer deflector and a
polarization management layer, doubling the apparent pixel density without reducing the original frame
rate. More recently, Neguyen et al., 2020 realized mechanical wobulation for both micro-OLED and LCD
panels to reduce the screen-door effect. These prior arts, based on the wobulation method, can simulate
high resolution imagery for the entire FOV before ideal display panels are available. Nonetheless, the
wobulation method still requires a large amount of data rate and cannot reduce the burden placed by the
massive amount of data flow.
Alternatively, the foveation approach aimed at local resolution enhancement can avoid this problem,
which makes use of the non-uniform angular resolution distribution of the human visual system (Rossi
and Roorda, 2010). It offers high resolution on the fovea region of eye retina while maintaining degraded
resolution on the peripherals. This principle was adopted for imaging before near-eye displays (Hua and
Liu, 2007). Generally, in most foveated VR systems, a beam splitter is employed to combine the images
displayed on the low-resolution panel and high-resolution one, resulting in a larger device volume.
Miniaturizing the optical layout and finding an alternative to the bulky beam splitter design is an essential
task for the future development of foveated VR devices. A promising candidate is using an off-axis mini-
projection unit together with a transparent projection screen on top of the display panel. The projection
screen should be transparent for the display light but manifest strong scattering for the off-axis projection
light. A decent example of such a projection screen is polymer-dispersed liquid crystal film with
customized molecular orientation and index mismatch (He et al., 2020). Moreover, since the gaze point is
not always fixed at the center FOV, another potential development direction for the foveation method is
image shifting, which is similar to but more complicated than beam steering technologies. Both
mechanical and optical shitting method for VR displays have been demonstrated, using a rotatable beam
splitter (Sahlsten, 2020) and a switchable liquid crystal deflector (Tan et al. 2018), respectively.
In parallel, a decent optical imaging part is also critical for generating high-resolution virtual images in
VR headsets. Due to ergonomic requirements, the viewing optics should be compact and lightweight,
which brings a significant sacrifice in imaging quality. Conventional aspheric singlet with smooth
surfaces usually have limited stray light but a large volume and weight. Thus, its compact Fresnel
alternative is more prevalent in current commercial VR headsets (Geng et al., 2018). Although Fresnel
singlets have more degrees of freedom for aberration control, its intrinsic diffractive artifacts and
unavoidable stray light considerably reduce the image sharpness. For now, the systematic imaging quality
is limited by the display panel resolution in most headsets, so these drawbacks of Fresnel lenses are still
tolerable. But in the long run, these issues could become more critical as display pixel density gradually
increases. To further reduce the device dimension, catadioptric pancake optics can be employed (Wong et
al., 2017). With reflective surfaces induced to share the optical power of refractive components, the
pancake lenses can allow display panel with smaller sizes due to their shorter focal length. However,
these benefits come at the cost of 75% light efficiency and demanding polarization control to eliminate
ghost images. In this case, plastic materials with limited birefringence and high-quality polarizers and
waveplates are highly demanded.
Moreover, the emerging flat optics including broadband diffractive lenses (Meem et al., 2020),
metalenses (Chen et al., 2019), and liquid crystal Pancharatnam-Berry phase lenses (Zhan et al., 2019)
can also be applied in the VR lens system for aberration control and system miniaturization. By adding a
thin-film flat polymer lens, it is possible to sharpen the imagery by more than three times (Zhan et al.,
2020). Another intriguing approach is to use a two-dimensional curved display (Grover et al., 2018). With
the field curvature compensated by the tailored panel curvature, the heavy burden on the lens design can
be well relieved. Alternatively, the curved fiber faceplate (Zhao et al., 2019) can be attached to the
display panel as a surface-shaping component, which can be designed together with the viewing optics for
sharper imaging.
Aside from limited resolution and screen-door effect, VAC is another significant issue in VR systems. A
plethora of solutions have been developed to mitigate this conflict (Kramida, 2015), but only few have
been applied to the current commercial VR headsets. Monovision displays represent a simple solution to
VAC, where vergence is not present for the virtual image. Since only one eye is offered with digital
images, this approach is more suitable for specific AR applications but not immersive VR. The other
extreme is accommodation-invariant approaches, like the Maxwellian view (Takaki and Fujimoto, 2018),
where the point source is focused on the pupil with angularly encoded amplitude information, and the
image on the retina is independent of the accommodation response. However, to tolerate the eye
movement, Maxwellian-view systems usually exhibit a limited FOV.
In general, most of other approaches offer a proper accommodation cue to mimic the retina blur and
therefore alleviate the conflict. A typical example is holographic display (Yamaguchi et al., 2007) aimed
at reconstructing accurate wavefront of the entire 3D scene and offering accurate retinal blur. Aside from
the limited FOV, holographic displays usually manifest degraded image quality due to laser speckles.
Similarly, light field displays (Wetzstein et al., 2012) reconstruct the geometric light rays instead of the
diffractive wavefront, which can also provide the approximately correct depth information and retina blur
but usually end up with a low resolution. If the amount of information is taken into consideration, it is not
surprising that these approaches aimed at showing volumetric information like holograms and light fields
cannot offer sufficient resolution with the limited bandwidth of current hardware. Even so, there is no
denying that these approaches may gradually mature in the long term with better hardware and eventually
become satisfactory for users.
In the short term, methods that can find an acceptable trade-off between depth accuracy and system
complexity should be more practical for addressing the VAC in current commercial products, such as
varifocal and multifocal displays. Varifocal displays employ an eye tracker to locate the gaze location and
an adaptive focusing component to shift the display depth accordingly. Additionally, real-time blur
rendering is also preferred in varifocal approaches because they cannot naturally generate retina blur
(Dunn et al., 2017). In comparison, multifocal displays (Liu and Hua, 2010; Hua, 2017; Zhan et al., 2018;
Tan et al., 2018; Liu et al., 2018) can create near-correct physical depth blur and offer a customizable
balance between depth accuracy and hardware bandwidth by choosing the density of focal planes for
different applications. A systematic summary and analysis of multifocal displays can be found in Zhan et
al., 2020. For both varifocal and multifocal displays, the need for high-quality focal changing components
is still urgent, which should have fast response time, compact form factor, and low power consumption.
Different from the immersive experience provided by VR, one of the most pressing challenges in AR is
expanding the FOV. Due to various designs and form factors for the same type AR, we will discuss and
compare the diagonal FOV instead of the horizontal/vertical FOV values. The diagonal FOV is related to
the horizontal/vertical FOV as 2 2
diagonal horizontal vertical
. To address the inadequate FOV issue,
we will overview potential solutions and analyze the systems case by case. In a lightguide-based near-eye
display (LNED), the light from optical engine propagates inside the lightguide following the TIR and is
then extracted to human eye by an exit pupil expansion (out-coupler) as illustrated in Figure 3A.
Typically, the core optical elements in such a system are the image source and the light combiner
consisting of an input coupler and an output coupler. The optical engine can be a liquid- crystal-on-silicon
(LCoS) panel, digital light processing (DLP), µOLED, µLED, and laser beam scanning (LBS) (Kress,
2020), while the combiners can be a reflective mirror or diffractive grating (Kress, 2019; Lee et al., 2019).
When the light propagating inside the lightguide, the TIR angle is governed by the refractive index of the
lightguide. Meanwhile, the index contrast of the coupler determines the angular and spectral responses,
especially for grating and hologram, which affects the color uniformity over the FOV and the eye-box
(Kress, 2019). Due to the significant impact of the coupler on the system, numerous technologies have
been applied to optimize the coupler performance (Xiang et al., 2018; Gao et al., 2017; Yin et al., 2019;
Yin et al., 2020). As a result, the angular response of a LNED system is not limited by the coupler but by
the critical angle of TIR, which is in turn determined by the lightguide refractive index. The normal
refractive index of lightguide is n
=1.50±0.03 (Sprengard et al., 2019), while a comparatively high
refractive index is n
=1.71.8 (Masuno et al., 2019). For most LNEDs, such as HoloLens 2 and Magic
Leap One, high index glass has been implemented to realizing a diagonal FOV of 50° (Kress, 2020). To
widen FOV further, a high index n
1.9 glass has been commercialized recently. By using such a high-
index glass, the critical angle becomes smaller so that the range from critical angle to 90° gets larger,
meaning a wider FOV can be supported in the lightguide.
In addition to improving the intrinsic characteristics of the components, such as increasing the refractive
index of glass or widening the angular bandwidth of coupler, the FOV can also be extended by expanding
the system’s degree of freedom. Through utilizing the multiplexing of coupler functions, such as spatial
multiplexing (Vallius et al., 2017), polarization multiplexing (Shi et al., 2018), etc., we can build a more
sophisticated system with wide FOV. The multiplexing method utilized for broadening FOV is essentially
to stitch images based on different characteristics of light, thereby realizing a more informative and
realistic experience. However, it is worth mentioning that the multiplexing is not limited in benefitting the
FOV, it also plays an essential role in overcoming the VAC issue (Zhan et al., 2019; He et al., 2020) and
presenting full-color images (Jang et al., 2017) in the AR system.
In a near-eye display, the multiplexing based on the properties of light can be categorized into spatial
multiplexing, time multiplexing, polarization multiplexing, wavelength multiplexing, and angular
multiplexing. Sometimes, more than one method is used in a system. By spatially combining two images
to increase the FOV, Microsoft patented a combiner structure with two intermediate couplers separated
spatially (Vallius et al., 2017). Then Shi et al. proposed the polarization multiplexing based on meta-
gratings (Shi et al., 2018). Similar to polarization division multiplexing in optical fiber communications
where two channels with orthogonal polarizations are used to double the information capacity, the
polarization multiplexing method increases the FOV by encoding the left and right FOVs into two
orthogonal polarization channels, TE and TM, respectively. Recently, Yoo et al. propose an extended
FOV LNED system by polarization multiplexing using LC-based grating (Yoo et al., 2020). In the
holographic volume grating (HVG)-based LNEDs, several multiplexing techniques have been reported.
Han et al. (2015) and Yu et al. (2017) attempted to apply the spatially multiplexing in out-coupler HVG to
obtain wide FOV. Lately, LC-based polarization volume gratings (PVGs), also known as Bragg
polarization gratings, with high diffraction efficiency and large angular bandwidth have been reported
(Lee et al., 2017; Yin et al., 2020). Due to these special optical features, it is feasible to build a spatially
multiplexed AR system with a large FOV using PVGs. As depicted in Figure 3B, the image information
is coupled into two lightguides through two input couplers that are spatially separated. Then the light
propagates into the output area through TIR, and the image information is extracted by two output
couplers with different periodicity and form a larger FOV beyond the limitation of lightguide TIR. Since
the asymmetric input and output coupler here may induce significant chromatic aberrations and image
distortion, it is preferable to employ narrow-band display engine and anamorphic image pre-processing.
The Maxwellian view is an observation method, in which the lens system forms an image of the light
source in the plane of the observer’s pupil, instead of looking at the source directly. Therefore, the effect
of the eye’s optical aberrations is minimized, and the quantity of light independent of pupil size is
increased (Westheimer, 1966; Sugawara et al., 2016). When applying this method in NEDs, the effective
eye pupil can be regarded as a tiny aperture, and the focal depth of the image will be dramatically
increased. Therefore, the system offers focus-free feature, i.e., no matter where the eye focuses, the image
is always clear. However, this method has its own limitations, especially the severely reduced eyebox. To
address this issue, Kim et al. (Kim et al., 2018) combined a Maxwellian view LNED with holographic
optical element (HOE) multiplexing to obtain an enlarged eyebox or a steering eyebox.
Figure 3C illustrates a typical schematic diagram of the Maxwellian view system. Based on geometric
optics, the Maxwellian view system can evolve into different forms, such as partially reflective elements
and LNEDs. From Figure 3C, the FOV of this system is directly related to the numerical aperture (NA) of
the lens system. With rapid technology development and urgent needs from industry, numerous novel flat
lenses with a wide acceptance angle and large aperture in both on-axis and off-axis types have emerged
(Khorasaninejad et al., 2016; Yin et al., 2020). Based on the HOE with a large NA, NVIDIA
demonstrated an 85° × 78° monocular FOV Maxwellian view system (Kim et al., 2019). Further efforts
have been investigated to enlarge the FOV. Xiong et al. 2020 demonstrated a large FOV AR system with
100° diagonal FOV by hybridizing the Maxwellian view and the lightguide-based exit pupil expander. By
increasing the NA and compressing the lens volume, both FOV and form factor of the Maxwellian-view
based NED system can be improved significantly.
For optical see-through AR displays, ACR is a critical parameter, which puts a strict requirement on
display luminance (Lee et al., 2019). As a general guideline, for indoor applications, the output luminance
of the AR display should be at least 500 nits. By contrast, for outdoor applications, the required
luminance would exceed 10,000 nits. To deliver such a high luminance, both microdisplay and efficient
relay/combiner optics are pivotal.
A roadmap of potential display engines is plotted in Figure 4. To provide a more general guideline on
how to choose display engines, a qualitative comparison among five candidates is summarized in Table 1.
Field-sequential LCoS is a reflective display based on polarization modulation of backlight (Huang et al.,
2018). Due to high brightness (10
to 10
nits) and commercial availability, it has been used in Magic
Leap One (Klug et al., 2016) and HoloLens (Kress SID 2017). A proper polarization conversion system
(PCS) can boost the efficiency and brightness of an LCoS since only light with a certain linear
polarization can be reflected by the polarization beam splitter (PBS) and modulated by the LCoS. In
traditional, large size LCoS projectors, a PCS consisting of a fly-eye lens, a PBS array, and a patterned
half-wave plate is integrated. However, as the form factor shrinks to microdisplay sizes, fabrication
difficulties and bulkiness of such a PCS has its limitation. Although some researchers proposed improved
PCSs based on thin-film polarization gratings (Kim et al., 2012; Du et al., 2015), the small form factor,
large angular bandwidth, and high efficiency are still lacking. Another fundamental issue of LCoS is its
limited dynamic range, as the relatively poor dark state will influence the see-through experience,
especially for indoor uses. A two-dimensional (2D) illumination or backlight with independently
addressable patches offers a promising solution, like the mini-LED array for LCD panels (Tan et al.,
2018). Similar to LCoS, DLP panels are field-sequential micromirror displays with high brightness
(Thompson et al., 2015), as employed by DigiLens. Compared to LCoS, the amplitude modulation of
DLP is polarization independent, and the dynamic range can be higher. For both reflective microdisplay
panels (LCoS and DLP), while LEDs are typically applied as the illumination source, other light sources,
such as lasers, are also available. Lasers are inherently collimated and linearly polarized and are very
suitable for LCoS. However, additional de-speckle optics are needed in order to achieve good image
In comparison with projection, emissive displays are less mature but have potential to reduce the form
factor. They exhibit intrinsically high dynamic range because of the true black state. Micro organic light-
emitting diode (μOLED) is a promising candidate for emissive microdisplays. The typical architecture is
patterned color filters on top of white OLEDs. To date, full-color μOLED displays with 3,000 to 5,000
nits in luminance and ~3,000 ppi (pixel per inch) in resolution have been achieved (Haas, 2018;
Motoyama et al., 2019). But for AR displays with a large eye-box, such brightness is still inadequate (Lee
et al., 2019). Future development should pay attention to boosting their brightness, device lifetime, and
current efficiency. On the other hand, micro light-emitting diode (μLED) is emerging and has potential to
become the next-generation display technology. The most recent development of 10-μm pitch (~1300 ppi)
full-color LED microdisplay has achieved 10
to 10
nits in luminance (Quesnel et al., 2020). Despite this
impressive progress, μLED still faces two major challenges. The first is to enhance the non-radiative
recombination when the area ratio of the side wall increases (Gou et al., 2019). This means, for small
μLED chips down to <5 μm, the external quantum efficiency would drop dramatically. The second issue
is how to realize full color and high resolution simultaneously, as mass transfer and assembly for such
tiny RGB LEDs is challenging (Lin et al., 2020; Wong et al., 2020). A parallel approach is to use blue
μLED to pump green and red quantum dots as color conversion (Huang et al., 2020). However, obtaining
a uniform, long lifetime color conversion layer without color crosstalk for such small pixel sizes is by no
means easy. Therefore, further effort is needed to develop mass transfer technique or color conversion
layer patterning technique for ultra-small pixel pitch (<5 µm) μLEDs.
As for scanning display systems, they are normally with high efficiency, small form factor, high dynamic
range, and high brightness using laser illumination. Typically, a 2D micro-electromechanical system
(MEMS) mirror or two 1D MEMS mirrors are applied to scan the laser beam in orthogonal directions to
form 2D images. Different from the panel-based displays, scanning displays do not have an object plane.
This unique property indicates that unlike panel-based displays form object images on the panel, the
scanning displays can directly form images on the retina. One prominent example is the laser beam
scanning system in North Focals (Alexander et al., 2018). As most scanning display engines have
intrinsically small exit pupil, they need a proper exit pupil expansion/steering, and thus the optical design
will be more sophisticated. In comparison with reflective and emissive displays, the image uniformity of
the scanning method is another inevitable issue that requires improvement.
The information generated from the optical engine will undergo magnifying optics and/or combiners and
finally project into human eyes. The combiners can be classified into two types: reflective and diffractive.
The reflective type includes freeform half mirrors, freeform prisms, birdbath combiners, and cascaded
mirrors (Wei et al., 2018; Cheng et al., 2011), while diffractive type covers all kinds of grating-coupler
based lightguide combiners and off-axis holographic optical element (HOE, not used in lightguide)
combiners (Li et al., 2016). Their schematic plots are shown in Figure 4, and a comparison among them is
illustrated in Table 2.
The freeform half mirrors, freeform prisms and birdbath combiners usually manifest decent imaging
quality and high optical efficiency, but mainly suffer from a large form factor. To reduce the form factor,
cascaded mirrors embedded in a lightguide has been invented. However, for lightguide combiners,
additional attention should be paid on see-through transmittance, see-through uniformity, stray light
control and image brightness uniformity. As a result, the image quality and optical efficiency are usually
compromised. The diffractive combiners are also introduced to reduce the form factor of traditional
reflective combiners. Different from the reflective counterpart, the chromatic nature of diffractive
elements needs to be considered in optical design. Off-axis HOEs combined with an LBS system can
provide a true glasses-like form factor yet a limited eye-box. To further enlarge the eye-box, grating-
coupled lightguide combiners are employed where the output coupler design is more complicated since it
can also perform as the exit pupil expander.
Currently, two types of gratings are employed in lightguide AR: holographic volume Bragg gratings
(VBGs) and surface relief gratings (SRGs). Due to the different refractive index contrast, they exhibit
different spectral and angular responses. The traditional VBGs with a small refractive index contrast
(δn0.05) manifest narrow spectral (~10 nm) and angular (~5° in air) bandwidths, while SRGs with a
large δn (0.5) show much broader spectral and angular bands (Lee et al., 2019). Interestingly, DigiLens
has developed a large δn VBG (close to LC birefringence) based on holographic polymer-dispersed liquid
crystal (HPDLC), which is switchable and it performs much better than traditional VBG (Brown et al.,
2018). Beside these two gratings, polarization volume gratings (PVGs) based on chiral liquid crystals
(CLCs) are also emerging (Yin et al., 2019). The refractive index contrast is essentially the birefringence
of the LC material and thus can be tuned within a broad range (from <0.1 to >0.4). As those grating
couplers are usually optimized for a particular polarization (e.g., a linear polarization for VBGs and SRGs,
and a circular polarization for PVGs), a PCS modulating the polarization of light from the display engine
and polarization management within the lightguide will be significant for improving the system efficiency.
Another unavoidable aspect of improving light efficiency is the 2D exit pupil expander (EPE) design.
Typically, a turn-around gradient-efficiency grating (also termed as fold grating) is performed to first
expand the eye-box in one direction within the lightguide. Then the output grating extends the eye-box in
another direction. Specifically, due to the inherent chromatic dispersion in diffraction, color uniformity
control is as challenging as brightness uniformity in most of the waveguide designs using diffractive
combiners. However, since there is a trade-off between optical efficiency of the gratings (both the turn-
around grating and the output grating) and color/brightness uniformity within the expanded eye-box,
finding an appropriate balance between them is essential from the system perspective.
In conclusion, we overviewed the major challenges and discussed potential opportunities of display optics
in the fast-developing field of AR and VR systems. The requirements from the human visual system are
analyzed in detail to offer quantitative standards for future near-eye display devices. These requirements
also bring out the major issues that need to be emphasized and addressed in current devices, regarding
panel resolution, form factor, imaging performance, VAC, FOV, and brightness. By learning from recent
advances in optics and developing trends of AR and VR devices, we shared a few thoughts about how to
meet these challenges in the near future and the long run.
This work was supported by Intel Corporation and GoerTek Electronics.
Author Contributions
Conceptualization, T.Z. and K.Y.; Methodology, T.Z. and K.Y.; Writing – Original Draft T.Z. K.Y. J.X.
and Z.H.; Writing – Review & Editing T.Z. K.Y. J.X. Z.H. and S.T.W; Supervision, S.T.W.
Declaration of Interests
The authors declare no competing interests.
Alexander, S., Bailey, M., Morrison, V.R., Holland, L.F., and Moore, J. (2018). Systems, devices, and
methods for eyebox expansion in wearable heads-up displays. U.S. Patent No. 9,989,764.
Allen, W. and Ulichney, R. (2005, May). 47.4: Invited paper: Wobulation: Doubling the addressed
resolution of projection displays. In SID Symposium Digest of Technical Papers (Vol. 36, No. 1, pp.
1514-1517). Oxford, UK: Blackwell Publishing Ltd.
Brown, R.D., Grant, A.J., Hendrick, W.L., Popovich, M.M., Stanley, J.H., and Waldern, J.D. (2018).
Transparent Waveguide Display. U.S. Patent Application No. 15/943,590.
Cakmakci, O. and Rolland, J. (2006). Head-worn displays: a review. J. Disp. Technol. 2, 199-216.
Chen, H., Lee, J. H., Lin, B.Y., Chen, S., and Wu, S.T. (2018). Liquid crystal display and organic light-
emitting diode display: present status and future perspectives. Light: Sci. & Appl. 7, 17168.
Chen, W.T., Zhu, A.Y., Sisler, J., Bharwani, Z. and Capasso, F. (2019). A broadband achromatic
polarization-insensitive metalens consisting of anisotropic nanostructures. Nature Communications 10,
Cheng, D., Wang, Y., Hua, H., and Sasian, J. (2011). Design of a wide-angle, lightweight head-mounted
display using free-form optics tiling. Opt. Lett. 36, 2098-2100.
Curcio, C.A., Sloan, K.R., Kalina, R.E., and Hendrickson, A.E. (1990). Human photoreceptor topography.
J. Comparative Neurol. 292, 497-523.
Du, T., Fan, F., Tam, A.M.W., Sun, J., Chigrinov, V.G., and Kwok, H.S. (2015). Complex nanoscale
ordered liquid crystal polymer film for high transmittance holographic polarizer. Adv. Mater. 27, 7191-
Dunn, D., Tippets, C., Torell, K., Kellnhofer, P., Akşit, K., Didyk, P., Myszkowski, K., Luebke, D. and
Fuchs, H. (2017). Wide field of view varifocal near-eye display using see-through deformable membrane
mirrors. IEEE transactions on visualization and computer graphics 23, 1322-1331.
Gao, K., McGinty, C., Payson, H., Berry, S., Vornehm, J., Finnemeyer, V., Roberts, B. and Bos, P. (2017).
High-efficiency large-angle Pancharatnam phase deflector based on dual-twist design. Opt. Express 25,
Geng, Y., Gollier, J., Wheelwright, B., Peng, F., Sulai, Y., Lewis, B., Chan, N., Lam, W.S.T., Fix, A.,
Lanman, D. and Fu, Y. (2018, May). Viewing optics for immersive near-eye displays: pupil swim/size
and weight/stray light. In Digital Optics for Immersive Displays (Vol. 10676, p. 1067606). International
Society for Optics and Photonics.
Gou, F., Hsiang, E.L., Tan, G., Chou, P. T., Li, Y. L. Lan, Y. F., and Wu, S. T. (2019). Angular color
shift of micro-LED displays. Opt. Express 27, A746-A756.
Grover, G., Salahieh, B. and Nestares, O. (2018, June). Curved screen virtual reality headsets. In 3D
Image Acquisition and Display: Technology, Perception and Applications (pp. 3M5G-3). Optical Society
of America.
Haas, G. (2018). Microdisplays for Augmented and Virtual Reality. SID Symp. Dig. Tech. Papers 49,
Han, J., Liu, J., Yao, X., and Wang, Y. (2015). Portable waveguide display system with a large field of
view by integrating freeform elements and volume holograms. Opt. Express 23, 3534-3549.
He, Z., Yin, K., and Wu, S.T. (2020). Passive polymer-dispersed liquid crystal enabled multi-focal plane
displays. Opt. Express 28, 15294-15299.
Hoefflinger, B. (2007). High-dynamic-range (HDR) vision. Springer Berlin Heidelberg.
Hoffman, D. M., Girshick, A. R., Akeley, K., and Banks, M. S. (2008). Vergence-accommodation
conflicts hinder visual performance and cause visual fatigue. J. Vision 8, 1–30.
Hua, H. (2017). Enabling focus cues in head-mounted displays. Proceedings of the IEEE 105, 805-824.
Hua, H. and Liu, S. (2008). Dual-sensor foveated imaging system. Appl. Opt. 47, 317-327.
Hua, H., Hu, X. and Gao, C. (2013). A high-resolution optical see-through head-mounted display with
eyetracking capability. Opt. Express 21, 30993-30998.
Huang, Y., Liao, E., Chen, R., and Wu, S.T. (2018). Liquid-crystal-on-silicon for augmented reality
displays. Appl. Sci. 8, 2366.
Huang, Y., Hsiang, E.L., Deng, M.Y., and Wu, S.T. (2020). Mini-LED, Micro-LED and OLED displays:
Present status and future perspectives. Light: Sci. & Appl. 9, 105.
Jang, C., Bang, K., Kim, J., Jeong, Y., and Lee, B. (2017). Full color virtual retinal display using a
holographic optical element. Imaging and Applied Optics (OSA) paper JTu5A.32.
Khorasaninejad, M., Chen, W. T., Devlin, R. C., Oh, J., Zhu, A. Y., and Capasso, F. (2016). Metalenses at
visible wavelengths: Diffraction-limited focusing and subwavelength resolution imaging. Science 352,
Kim, J., Jeong, Y., Stengel, M., Akşit, K., Albert, R., Boudaoud, B, Greer, T., Kim, J., Lopes, W.,
Majercik, Z., et al. (2019). Foveated AR: dynamically-foveated augmented reality display. ACM
Transactions on Graphics (TOG) 38, 1-15.
Kim, J., Jeong, Y., Stengel, M., Akşit, K., Albert, R., Boudaoud, B., Greer, T., Kim, J., Lopes, W.,
Majercik, Z. and Shirley, P. (2019). Foveated AR: dynamically-foveated augmented reality display. ACM
Trans. Graph. 38, 1-15.
Kim, J., Komanduri, R.K., Lawler, K.F., Kekas, D.J., and Escuti, M.J. (2012). Efficient and monolithic
polarization conversion system based on a polarization grating. Appl. Opt. 51, 4852-4857.
Kim, S.B. and Park, J.H. (2018). Optical see-through Maxwellian near-to-eye display with an enlarged
eyebox. Opt. Lett. 43, 767-770.
Klug, M.A., Cahall, S.C., and Chung, H. (2016). Separated pupil optical systems for virtual and
augmented reality and methods for displaying images using same. U.S. Patent Application No.
Kramida, G. (2015). Resolving the vergence-accommodation conflict in head-mounted displays. IEEE
Transactions on Visualization and Computer Graphics 22, 1912-1931.
Kress, B.C. (2019, July). Optical waveguide combiners for AR headsets: features and limitations. In
Digital Optical Technologies 2019 (Vol. 11062, p. 110620J). International Society for Optics and
Kress, B.C. (2020). Optical Architectures for Augmented-, Virtual-, and Mixed-Reality Headsets.
Kress, B.C. and Cummings, W.J. (2017, May). 111: Invited paper: Towards the ultimate mixed reality
experience: HoloLens display architecture choices. In SID symposium digest of technical papers (Vol. 48,
No. 1, pp. 127-131).
Lee, Y.H., He, Z., and Wu, S.T., (2019). Optical properties of reflective liquid crystal polarization volume
gratings. J. Opt. Soc. Am. B 36, D9-D12.
Lee, Y.H., Yin, K., and Wu, S.T. (2017). Reflective polarization volume gratings for high efficiency
waveguide-coupling augmented reality displays. Opt. Express 25, 27008-27014.
Lee, Y.H., Zhan, T., and Wu, S.T. (2017). Enhancing the resolution of a near-eye display with a
Pancharatnam–Berry phase deflector. Opt. Lett. 42, 4732-4735.
Lee, Y.H., Zhan, T., and Wu, S.T. (2019). Prospects and challenges in augmented reality displays. Virtual
Reality & Intelligent Hardware 1, 10-20.
Li, G., Lee, D., Jeong, Y., Cho, J., and Lee, B. (2016). Holographic display for see-through augmented
reality using mirror-lens holographic optical element. Opt. Lett. 41, 2486-2489.
Lin, J.Y. and Jiang, H.X. (2020). Development of microLED. Appl. Phys. Lett. 116, 100502.
Liu, S. and Hua, H. (2010). A systematic method for designing depth-fused multi-focal plane three-
dimensional displays. Opt. Express 18, 11562-11573.
Liu, S., Li, Y., Zhou, P., Chen, Q., Li, S., Liu, Y., Wang, Y. and Su, Y. (2018). Fullcolor multiplane
optical seethrough headmounted display for augmented reality applications. J. Soc. Inf. Disp. 26, 687-
Masuno, A., Iwata, T., Yanaba, Y., Sasaki, S., Inoue, H., and Watanabe, Y. (2019). High refractive index
La-rich lanthanum borate glasses composed of isolated BO
units. Dalton Trans. 48, 10804-10811.
Meem, M., Banerji, S., Pies, C., Oberbiermann, T., Majumder, A., Sensale-Rodriguez, B. and Menon, R.
(2020). Large-area, high-numerical-aperture multi-level diffractive lens via inverse design. Optica 7, 252-
Motoyama, Y., Sugiyama, K., Tanaka, H., Tsuchioka, H., Matsusaki, K., and Fukumoto, H. (2019).
High efficiency OLED microdisplay with microlens array. J. Soc. Inf. Disp. 27, 354-360.
Nguyen, J., Smith, C., Magoz, Z. and Sears, J. (2020, February). Screen door effect reduction using
mechanical shifting for virtual reality displays. In Optical Architectures for Displays and Sensing in
Augmented, Virtual, and Mixed Reality (AR, VR, MR) (Vol. 11310, p. 113100P). International Society
for Optics and Photonics.
Quesnel, E., Lagrange, A., Vigier, M., Consonni, M., Tournaire, M., Le Marchand, V., Suhm, A., Demars,
P., Pillet, J.-C., Ben Bakir, B., Olivier, N., Feltin, E., Lamy, J.M., D’Amico, M., Cao, E., Haas, G.,
Charrier, L., and Coni, P. (2020). Dimensioning a full color LED microdisplay for augmented reality
headset in a very bright environment. J. Soc. Inf. Disp. DOI:10.1002/jsid.884.
Rossi, E. A. and Roorda, A. (2010). The relationship between visual resolution and cone spacing in the
human fovea. Nature Neurosci. 13, 156-157.
Sahlsten, O. (2020, April). Immersive Mixed Reality (Conference Presentation). In Optical Architectures
for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) (Vol. 11310, p.
1131024). International Society for Optics and Photonics.
Shi, Z., Chen, W.T., and Capasso, F. (2018, May). Wide field-of-view waveguide displays enabled by
polarization-dependent metagratings. In Digital Optics for Immersive Displays (Vol. 10676, p. 1067615).
International Society for Optics and Photonics.
Sprengard, R., Sjogren, B., Nass, P., Ottermann, C., Lange, B., Damm, T., Plapper, V., Dietrich, V.,
Maurer, U., Fang, H. and Weng, N. (2019). High Refractive Index Glass Wafers for Augmented Reality
Devices using Waveguide Technology: Recent Advances in Control of Quality Parameters and their
Correlation with Device Properties. SID Symp. Dig. Tech. Pap. 50, 116-120.
Sugawara, M., Suzuki, M., and Miyauchi, N. (2016). Retinal Imaging Laser Eyewear with FocusFree
and Augmented Reality. SID Symp. Dig. Tech. Pap. 47, 164-167.
Takaki, Y. and Fujimoto, N. (2018). Flexible retinal image formation by holographic Maxwellian-view
display. Opt. Express 26, 22985-22999.
Tan, G., Huang, Y., Li, M.C., Lee, S.L., and Wu, S.T. (2018). High dynamic range liquid crystal displays
with a mini-LED backlight. Opt. Express 26, 16572-16584.
Tan, G., Lee. Y. H., Zhan. T., Yang, J., Liu, S., Zhao, D., and Wu, S. T. (2018). Foveated imaging for
near-eye displays. Opt. Express 26. 25076-25085.
Tan, G., Zhan, T., Lee, Y. H., Xiong, J., and Wu, S.T. (2018). Polarization-multiplexed multi-plane
display. Opt. Lett. 43, 5651-5654.
Thompson, J., Pettitt, G., and Ferri, J. (2015). Practical Application of TI DLP® Technology in the Next
Generation Head-up Display System. SID Symp. Dig. Tech. Papers 46, 700–703.
Vallius, T., and Tervo, J. (2017). U.S. Patent No. 9,791,703. Washington, DC: U.S. Patent and Trademark
Watt, S. J., Akeley, K., Ernst, M. O., and Banks, M. S. (2005). Focus cues affect perceived depth. J.
Vision 5, 7-7.
Wei, L., Li, Y., Jing, J., Feng, L., and Zhou, J. (2018). Design and fabrication of a compact off-axis see-
through head-mounted display using a freeform surface. Opt. Express 26, 8550-8565.
Westheimer, G. (1966). The maxwellian view. Vision research 6, 669-682.
Wetzstein, G., Lanman, D., Hirsch, M. and Raskar, R. (2012). Tensor displays: compressive light field
synthesis using multilayer displays with directional backlighting. ACM Transactions on Graphics 31, 1-
Wheatstone, C. (1838). XVIII. Contributions to the physiology of vision. Part the first. On some
remarkable, and hitherto unobserved, phenomena of binocular vision. Philosophical transactions of the
Royal Society of London, 128, 371-394.
Wheelwright, B., Sulai, Y., Geng, Y., Luanava, S., Choi, S., Gao, W., & Gollier, J. (2018, May). Field of
view: not just a number. In Digital Optics for Immersive Displays (Vol. 10676, p. 1067604). International
Society for Optics and Photonics.
Wong, M.S., Nakamura, S., and DenBaars, S.P. (2020). Progress in high performance III-Nitride micro-
light-emitting diodes. ECS J. Solid State Sci. Technol. 9, 015012.
Wong, T.L., Yun, Z., Ambur, G. and Etter, J. (2017, June). Folded optics with birefringent reflective
polarizers. In Digital Optical Technologies 2017 (Vol. 10335, p. 103350E). International Society for
Optics and Photonics.
Xiang, X., Kim, J., and Escuti, M.J. (2018). Bragg polarization gratings for wide angular bandwidth and
high efficiency at steep deflection angles. Sci. Rep. 8, 7202.
Xiong, J., Tan, G., Zhan, T. and Wu, S.T. (2020). Wideview augmented reality display with diffractive
cholesteric liquid crystal lens array. J. Soc. Inf. Disp. 28, 450-456.
Yamaguchi, T., Okabe, G. and Yoshikawa, H. (2007). Real-time image plane full-color and full-parallax
holographic video display system. Opt. Eng. 46, 125801.
Yin, K., He, Z., and Wu, S.T. (2020). Reflective Polarization Volume Lens with Small f Number and
Large Diffraction Angle. Adv. Opt. Mater. 8, 2000170.
Yin, K., Lin, H.Y., and Wu, S.T. (2019). Chirped polarization volume grating with ultra-wide angular
bandwidth and high efficiency for see-through near-eye displays. Opt. Express 27, 35895-35902.
Yin, K., Lin, H.Y., and Wu, S.T. (2020). Chirped polarization volume grating for wide FOV and high
efficiency waveguide-based AR displays. J. Soc. Inf. Disp. 28, 368-374.
Yoo, C., Bang, K., Chae, M., and Lee, B. (2020). Extended-viewing-angle waveguide near-eye display
with a polarization-dependent steering combiner. Opt. Lett. 45, 2870-2873.
Yu, C., Peng, Y., Zhao, Q., Li, H., and Liu, X. (2017). Highly efficient waveguide display with space-
variant volume holographic gratings. Appl. Opt. 56, 9390-9397.
Zhan, T., Lee, Y.H. and Wu, S.T. (2018). High-resolution additive light field near-eye display by
switchable Pancharatnam–Berry phase lenses. Opt. Express 26, 4863-4872.
Zhan, T., Lee, Y.H., Tan, G., Xiong, J., Yin, K., Gou, F., Zou, J., Zhang, N., Zhao, D., Yang, J. and Liu, S.
(2019). Pancharatnam–Berry optical elements for head-up and near-eye displays. J. Opt. Soc. Am. B 36,
Zhan, T., Xiong, J., Tan, G., Lee, Y.H., Yang, J., Liu, S. and Wu, S.T. (2019). Improving near-eye
display resolution by polarization multiplexing. Opt. Express 27, 15327-15334.
Zhan, T., Xiong, J., Zou, J. and Wu, S.T. (2020). Multifocal displays: review and prospect. PhotoniX 1,
Zhan, T., Zou, J., Lu, M., Chen, E., and Wu, S. T. (2019). Wavelength-multiplexed multi-focal-plane
seethrough near-eye displays. Opt Express 27, 27507-27513.
Zhan, T., Zou, J., Xiong, J., Liu, X., Chen, H., Yang, J., Liu, S., Dong, Y. and Wu, S.T. (2020). Practical
Chromatic Aberration Correction in Virtual Reality Displays Enabled by CostEffective UltraBroadband
Liquid Crystal Polymer Lenses. Advanced Optical Materials 8, 1901360.
Zhao, R., Zhou, Y., Wei, D., Wang, Y., Fu, Y. and Wang, J. (2019, December). A high-resolution optical-
fiber imaging sensor. In AOPC 2019: Optical Fiber Sensors and Communication (Vol. 11340, p.
1134007). International Society for Optics and Photonics.
Figure 1. Illustration on the performance of human vision. (A) The profile of human FOV. (B) The
relation between human visual acuity and visual angle. (C) Sketch of the VAC issue. The accommodation
cue coincides with vergence cue when viewing a real object (left). The mismatch occurs when viewing a
virtual object displayed at a fixed plane (right).
Figure 2. The development trend of panel resolution. The pixel density of display panels will gradually
increase for VR application. Before panels with ideal pixel density are available at low cost, it is also
feasible to employ global resolution enhancement based on mechanical or optical wobulation method and
local resolution enhancement with foveated display technologies.
Figure 3. Optical structures of AR systems with extended FOV. (A) Schematic illustration of the
LNED system. TIR happens at each reflection during the propagation, and the angle is marked in orange.
(B) Lightguide-based polarization multiplexing system for enlarging FOV. The system is based on two
PVGs with opposite polarization responses (LCP and RCP) and different diffraction angles. (C)
Schematic diagrams of the Maxwellian view system, including the imaging principle and two distinct
forms derived from it, partial reflector and lightguide structure.
Figure 4. Schematic plots of major microdisplays and combiners. The microdisplays cover liquid-
crystal-on-silicon (LCoS), digital light processer (DLP), laser beam scanner (LBS), micro organic light-
emitting diode (μOLED), and micro light-emitting diode (μLED), while the combiners include freeform
half mirror, birdbath, freeform prism, off-axis holographic optical element (HOE), cascaded mirrors, and
grating couplers. Three kinds of grating couplers are also highlighted: surface relief grating (SRG),
volume Bragg grating (VBG), and polarization volume grating (PVG).
Table 1. Comparison among AR display light engines.
Display Maturity
efficiency Form factor
Optical system
Table 2. Comparisons among AR optical combiners.
Type Combiner Efficiency*
Form factor
δn Bandwidth FOV diagonal**
Freeform mirror
Freeform prism
Cascaded mirrors
axis HOE
Traditional VBG
*These typical values depend on lightguide design.
**These typical values come from products and prototypes.
Introducing the fundamentals of emerging augmented reality (AR) and virtual reality (VR)
technologies and their potential applications
Analyzing the technical challenges of AR and VR displays
Presenting potential solutions to overcome these challenges case by case
... Despite the advanced stage of development in 2D display technology, achieving natural and vivid 3D displays remains the ultimate goal [4][5][6]. More recently, virtual reality (VR) and augmented reality (AR) have emerged as the most promising candidates for the next generation of mobile platforms, facilitating the advancement of 3D display technology [7][8][9]. Notably, Apple's introduction of the groundbreaking Apple Vision Pro headset in June 2023 has ignited researchers' enthusiasm towards AR/VR displays and 3D display technologies. ...
Full-text available
In recent years, the emergence of virtual reality (VR) and augmented reality (AR) has revolutionized the way we interact with the world, leading to significant advancements in 3D display technology. However, some of the currently employed 3D display techniques rely on stereoscopic 3D display method, which may lead to visual discomfort due to the vergence-accommodation conflict. To address this issue, several true 3D technologies have been proposed as alternatives, including multi-plane displays, holographic displays, super multi-view displays, and integrated imaging displays. In this review, we focus on planar liquid crystal (LC) devices for different types of true 3D display applications. Given the excellent optical performance of the LC devices, we believe that LC devices hold great potential for true 3D displays.
... The eyepiece can consist of a group of refractive or diffractive optics, and the system configuration can be either transmissive, as shown in Fig. 2(a), or reflective with a folded optical path for a more compact form factor, as in a pancake scheme 81,82 . High-resolution liquid crystal displays (LCDs), organic light-emitting-diode (OLED) microdisplays, and micro light-emitting-diode (μLED) microdisplays are commonly employed as image sources 83 in VR displays. ...
... Nevertheless, owing to the refractive index mismatch with the surrounding air, the surface reflection by the relief structure significantly hinders the development of Fresnel lenses from only being appealing to being undisputed in such applications. For example, the effect of a 'ghost image' or veil glares [9][10][11] caused by the reflection of Fresnel lenses in an augmented reality projector significantly disturbs the consumer experience. More confusing and intractable, typical anti-reflection interference films [12,13] are nearly invalid for the Fresnel lenses because the optical path difference between the interference light beams in different regions is not always the same. ...
Full-text available
Nanopatterning complex uneven surface of numerous functional devices to improve their performance is significantly appealing; however, it is extremely challenging. This study proposes a discretely-supported transfer nanoimprint technique to fabricate nanostructures on complex device surfaces containing multi-spatial frequencies. First, a discretely-supported nanoimprint template was designed based on the built energy criterion. A contact fidelity of over 99% was achieved between the designed template and the targeted complex uneven substrate surface. Next, the prefilled nanostructures on the template were transferred to the target surface after contact. By precisely controlling the amount of micro-droplet jetting on the template on-demand, the accumulation of the polymer in the micro-valley sites on the complex substrate was avoided, thus maintaining the morphology and generating function of the devices. Finally, high-quality Fresnel lenses with broadband wide-directional antireflection and excellent imaging performance were developed by imprinting subwavelength-tapered nanostructures on the relief surface.
... Immersive virtual reality (VR) has grown rapidly over the past several years and is widely used in many areas of life and science. It is used in healthcare, education, engineering, manufacturing, and entertainment, among others [1][2][3][4][5]. There is also a growing interest in using this technology in athletic training to diagnose and control athletes' motor skills and improve their sports performance [6][7][8][9]. ...
Full-text available
Active virtual reality games (AVRGs) have become more and more popular. As the intensity of this form of physical activity (PA) may be insufficient to achieve health-related benefits, it is worth looking for solutions that increase the intensity of PA. The main aim of the study was to evaluate the effect of leg loading in the form of ankle weights (AWs) on the PA intensity of young adults playing AVRGs using an omnidirectional treadmill. The enjoyment of the game and users' perceptions of the usefulness of this type of exercise were also evaluated. The study involved 26 university students. Each participant played an AVRG game on an omnidirectional treadmill twice, without and with ankle weights (2 kg per leg). The intensity of PA was evaluated using a heart rate monitor. The attractiveness of the game was assessed using the Physical Activity Enjoyment Scale (PACES). The study found that the percentage of maximum heart rate in participants playing AVRGs without ankle weights was significantly lower than that observed when playing with the weights. In both cases, PA intensity was high. A survey showed that the weights attached to the ankles did not affect the perceptions of the enjoyment of the game. The use of ankle weights appears to be an effective and simple way to increase the intensity of physical exercise during AVRGs based on locomotor movements performed with the lower limbs, especially since, according to study participants, such a procedure does not negatively affect the enjoyment of the game. Due to the high intensity of PA while playing VR games using an omnidirectional treadmill, it can be assumed that regular use of this solution is likely to provide health benefits.
Light‐emitting diodes (LEDs) producing pure, highly saturated colors are the industry standard for efficient backlighting of high‐color gamut displays. Vivid color reproduction, matching the eye's perception of nature, is the central paradigm in the design of narrow‐band emitting phosphors. To cover a wide range of naturally occurring color tones, expansion of the color gamut in the green spectral region, and therefore an advanced applicable green phosphor, is highly desired. Herein, the oxonitridoberyllosilicate Ba[BeSiON 2 ]:Eu ²⁺ showing outstanding narrow‐band green emission (λ max ≈526 nm with FWHM ≈1600 cm ⁻¹ (≈45 nm), x = 0.212, y = 0.715) when excited with InGaN‐based blue LEDs is presented. High quantum efficiency and low thermal quenching (>90% rel. quantum efficiency at 100 °C) as well as excellent scalability make the material suitable for industrial application in high color‐gamut LED displays. A prototype phosphor‐converted‐LED (pc‐LED), with green‐emitting Ba[BeSiON 2 ]:Eu ²⁺ and K 2 SiF 6 :Mn ⁴⁺ as red phosphor shows an extraordinary coverage in the CIE 1931 color space of 109% compared to the DCI‐P3 standard, topping the widely applied β ‐SiAlON:Eu ²⁺ phosphor (104%), making it suitable for use in phone displays, monitors, and television screens.
In GaN-based vertical micro LEDs, conventional metal n-contacts on the N face n-GaN suffer from a low aperture ratio due to the high reflection of metals, resulting in low-light extraction efficiencies. Great efforts have been devoted to enhancing transparency by employing transparent conducting oxides for n-contacts, but they exhibited poor Ohmic behavior due to their large work functions. Herein, we introduce an InN/ITO n-contact to achieve both superior contact property and high transparency. At the initial stage, the ITO with thin In interlayer was utilized, and the change in contact properties was observed with different annealing temperatures in the N 2 atmosphere. After annealing at 200 °C, the In/ITO n-contact exhibited Ohmic behavior with high a transparency of 74% in the blue wavelength region. The metallic In transformed into InN during the annealing process, as confirmed by transmission electron microscopy. The formation of InN caused polarization-induced band bending at the InN/GaN interface, providing evidence of enhanced Ohmic properties. In the application of vertical GaN µLED, the EQE increased from 6.59% to 11.5% while operating at 50 A/cm ² after the annealing process.
Immersive learning using fully immersive virtual reality (VR) is typically utilized to present 3D objects that cannot be represented easily on a computer screen. In the last few decades, the use of e-books to visualize documents has become prevalent. However, transferring an original document designed to be printed requires extraction of the document’s logical structure. The document must be broken down into fragments of text and images with the extraction of its logical structure. The fragments are then recomposed together to be presented as an e-book. In this paper, we propose a novel paradigm where documents can be presented in their original form without the need to be physically restructured, thanks to VR. With a VR headset, the proposed document visualizer allows users to access, display, and annotate documents. Such a system is particularly relevant for students and scholars in the humanities who investigate ancient documents where the distinct elements composing the document are intertwined. Furthermore, VR technology enhances the accessibility and equity of learning by offering various input modalities.
Full-text available
Presently, liquid crystal displays (LCDs) and organic light-emitting diode (OLED) displays are two dominant flat panel display technologies. Recently, inorganic mini-LEDs (mLEDs) and micro-LEDs (μLEDs) have emerged by significantly enhancing the dynamic range of LCDs or as sunlight readable emissive displays. “mLED, OLED, or μLED: who wins?” is a heated debatable question. In this review, we conduct a comprehensive analysis on the material properties, device structures, and performance of mLED/μLED/OLED emissive displays and mLED backlit LCDs. We evaluate the power consumption and ambient contrast ratio of each display in depth and systematically compare the motion picture response time, dynamic range, and adaptability to flexible/transparent displays. The pros and cons of mLED, OLED, and μLED displays are analysed, and their future perspectives are discussed. Mini and micro light-emitting diodes (LEDs) could move to the centre-stage of display screen technologies once they mature. Shin-Tson Wu of the University of Central Florida and colleagues analysed the pros, cons, and future prospects of the latest display screen technologies, especially for use in smartphones, smart watches, virtual and augmented reality, and heads-up vehicle displays. These applications require bright, flexible, transparent, and power-efficient displays. The currently dominant liquid crystal displays (LCDs) require a backlight unit, dictating their shape and flexibility. LCDs with a backlight unit made from mini LEDs are becoming rapid contenders to the conventional technology. So are displays using organic light-emitting diodes, but these are limited in their brightness and lifespans. Emissive displays made from mini and micro-LEDs show huge potential once manufacturing costs can be brought down.
Full-text available
A multi-focal plane see-through near-eye display using a transparent projection display is demonstrated. The key component of the transparent projection display is a passive polymer-dispersed liquid crystal (PDLC), which is highly transparent for a large range of incident angles in air but strongly scattering at large oblique angles in high refractive index medium (e.g. glass). The use of a passive device can avoid temporal multiplexing. Such a display is highly transparent in air and can easily deliver full-color images. The proposed method is an important step toward transparent display-enabled multi-focal plane displays.
Full-text available
This paper focuses on the dimensioning of a very bright full color 10 μm‐pitch light‐emitting device (LED) microdisplay for avionics application. Starting from the specifications of head‐mounted display to be used in an augmented reality optical system, a theoretical approach is proposed that enables predicting the specifications of the main technology building blocks entering into the microdisplay manufacturing process flow. By taking into account various material and technological parameters, kept as realistic as possible, it is possible to assess the feasibility of a very bright LED microdisplay (1 Mcd/m2 full white) and to point out the main limitations. The theoretical specifications are then compared with the technical results obtained so far in the framework of the H2020 Clean Sky “HILICO” project. It shows that 350 000 cd/m2 of white emission may be accessible with the present gallium nitride (GaN)‐micro‐LED technology provided a color conversion solution with stable external quantum efficiency of 30% is available. Beyond such level of luminance, the inherent limitations of driving circuit (4 V, 15 μA per pixel) commands working with materials enabling higher external quantum efficiency (EQE). In particular, 10‐μm‐pitch micro‐LEDs with electroluminescence EQE of 15% and color conversion EQE approaching 60% are needed, opening the way to future challenging material and technology research developments. Head‐mounted display for avionics requires particularly bright displays, challenging manufacturers. For that reason, we developed a dedicated approach to predict the specifications of technology building blocks entering into the display manufacturing process flow. As a result, a realistic feasibility assessment could be done based on the current technology capabilities including CMOS driving integrated circuit, micro‐LED manufacturing, and color conversion.
Full-text available
Conventional stereoscopic three-dimensional displays suffer from vergence- accommodation conflict because the stimulus to accommodation is fixed by the display panel and viewing optics, but that to vergence changes with image contents. With the recent rapid development of head-mounted displays, several methods have been proposed to offer the accommodation cues, among which multifocal display technology is an effective and practical solution. The first two decades of this century has witnessed the fast growth of multifocal displays from basic concept to mature implementations. This review systematically presents the state-of-the-art multifocal display design and development. Firstly, a comprehensive classification of numerous potential optical architectures to provide the multiplanar functionality is introduced, based on how the information is multiplexed and how the focal planes are generated. Next, the strengths and obstacles of reported or potential designs in each category are analyzed and compared with each other. In addition to enabling optics, the image rendering approaches for the multifocal planes are also described. This review presents a sufficient collection of past designs and is expected to offer a roadmap for future research and development of multifocal displays.
Full-text available
Planar optics based on patterned cholesteric liquid crystals (CLCs) has attracted increasing attention owing to the self‐organized helical structure and the ability to create arbitrary reflected wavefront through spatial orientation control. However, because of the subwavelength‐orientation requirement, it is challenging for liquid crystal lens to achieve a low f‐number (f/#) and large deflection angle simultaneously. Furthermore, with the increasing demand for compact size in novel optical systems, reflective lens has advantage over the transmissive one because it can fold the optical path. Here, a new off‐axis reflective polarization volume lens (PVL) with f/# = 0.825, large aperture size, simple fabrication process, thin profile, circular polarization selectivity, and large diffraction angle is proposed. In contrast to traditional vertical spiral structure, PVL is based on patterned CLCs with a slanted helical axis. In this paper, the PVL is theoretically evaluated and then three reflective PVLs at red, green, and blue wavelengths (R = 605 nm, G = 532 nm, and B = 450 nm) are fabricated. Meanwhile, a simple approach is utilized to achieve 20 mm diameter and 16.5 mm focal length. The low scattering and good image quality of reflective PVL enrich these functional devices and provide promising applications to novel foldable optical systems and waveguide‐based wearable near‐eye displays.
A novel augmented reality display is proposed to achieve a field of view of 100°, while maintaining a good form factor with a glass‐thin waveguide combiner. The out‐coupler consists of an array of off‐axis diffractive lenslets with extremely low f‐number. A breadboard system is built and its performance characterized, combined with analysis and discussion of further improvement. A wide‐view see‐through display with good form factor and large eyebox is proposed. The out‐coupler, a diffractive cholesteric liquid crystal lens array with extremely low f‐number, is fabricated using polarization holography method. The system is built and characterized, combined with analysis and discussion of further improvement.
A waveguide-based near-eye display (WNED) with an extended viewing angle using a polarization-dependent steering combiner (PDSC) is proposed. The novel eyepiece-combiner is composed of polarization gratings and polarization optics attached to the outcoupler part of the waveguide, which can control the output beam path depending on the polarization state. The viewing angle limited by the grating properties can be extended up to twice. Also, an ultrathinness of about 1.4 mm is suitable for the WNED. The demonstrated prototype system achieves a horizontal field of view of 33.2°, which is 2 times wider than the conventional structure (without the PDSC). The proposed configuration can resolve the viewing angle issue for the WNED.
We demonstrate a reflective chirped polarization volume grating as a strong contender for wide field‐of‐view augmented reality display systems. By introducing gradient pitch along the beam propagation direction, the angular bandwidth extends dramatically from 18° to 54° while keeping over 80% average efficiency and 95% peak efficiency. We fabricate a chirped polarization volume grating (CPVG) by introducing gradient pitch structure to conventional PVG. The angular bandwidth of CPVG is dramatically extended from 17° to 54° while keeping over 80% diffraction efficiency. Due to its wide angular bandwidth, high efficiency, low scattering, high transmittance, and simple fabrication process, our CPVG would be a strong contender for see‐through near‐eye displays.
This perspective provides an overview of early developments, current status, and remaining challenges of microLED (μLED) technology, which was first reported in Applied Physics Letters in 2000 [S. X. Jin, J. Li, J. Z. Li, J. Y. Lin and H. X. Jiang, "GaN Microdisk Light Emitting Diodes," Appl. Phys. Lett. 76, 631 (2000)]. Today, microLED is recognized as the ultimate display technology and is one of the fastest-growing technologies in the world as technology giants utilize it on a wide range of products from large flat panel displays and televisions, wearable displays, and virtual reality displays to light sources for the neural interface and optogenetics. It is anticipated that the collective R&D efforts worldwide will bring microLED products not only to the mass consumer electronic markets but also to serve the society on the broadest scale by encompassing sectors in medical/health, energy, transportation, communications, and entertainment.