Augmented Reality and Virtual Reality Displays: Perspectives and Challenges

Abstract

As one of the most promising candidates for next-generation mobile platform, augmented reality (AR) and virtual reality (VR) have potential to revolutionize the ways we perceive and interact with various digital information. In the meantime, recent advances in display and optical technologies, together with the rapidly developing digital processers, offer new development directions to advancing the near-eye display systems further. In this perspective paper, we start by analyzing the optical requirements in near-eye displays poised by the human visual system and then compare it against the specifications of state-of-the-art devices, which reasonably shows the main challenges in near-eye displays at present stage. Afterward, potential solutions to address these challenges in both AR and VR displays are presented case by case, including the most recent optical research and development that are already or have the potential to be industrialized for extended reality displays.

Full text

Journal Pre-proof
Augmented reality and virtual reality displays: Perspectives and challenges
Tao Zhan, Kun Yin, Jianghao Xiong, Ziqian He, Shin-Tson Wu
PII: S2589-0042(20)30585-X
DOI: https://doi.org/10.1016/j.isci.2020.101397
Reference: ISCI 101397
To appear in: ISCIENCE
Please cite this article as: Zhan, T., Yin, K., Xiong, J., He, Z., Wu, S.-T., Augmented reality and
virtual reality displays: Perspectives and challenges, ISCIENCE (2020), doi: https://doi.org/10.1016/
j.isci.2020.101397.
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition
of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of
record. This version will undergo additional copyediting, typesetting and review before it is published
in its final form, but we are providing this version to give early visibility of the article. Please note that,
during the production process, errors may be discovered which could affect the content, and all legal
disclaimers that apply to the journal pertain.
© 2020 The Author(s).

Augmented reality and virtual reality displays:
Perspectives and challenges
Tao Zhan,
1,2
Kun Yin,
1,2
Jianghao Xiong,
1
Ziqian He,
1
and Shin-Tson Wu
1,*
1
College of Optics and Photonics, University of Central Florida, Orlando, FL 32816, USA
2
These authors contributed equally
*Correspondence: swu@creol.ucf.edu
SUMMARY
As one of the most promising candidates for next-generation mobile platform, augmented reality (AR)
and virtual reality (VR) have potential to revolutionize the ways we perceive and interact with various
digital information. In the meantime, recent advances in display and optical technologies, together with
the rapidly developing digital processers, offer new development directions to advancing the near-eye
display systems further. In this perspective paper, we start by analyzing the optical requirements in near-
eye displays poised by the human visual system and then compare it against the specifications of state-of-
the-art devices, which reasonably shows the main challenges in near-eye displays at present stage.
Afterward, potential solutions to address these challenges in both AR and VR displays are presented case
by case, including the most recent optical research and development that are already or have the potential
to be industrialized for extended reality displays.
INTRODUCTION
As the most critical information acquisition medium, information displays have been developing rapidly
after the third industrial revolution. From the beginning of this millennium, display technologies have
successfully evolved from the bulky cathode ray tube to compact flat panel designs, such as liquid crystal
display (LCD) and organic light-emitting diode (OLED) (Chen et al., 2018). More recently, the next-
generation display technologies under dedicated development are no longer limited to flat panels that just
placed in front of the users but aimed at revolutionizing the way of interactions between the users and
their surrounding environment (Cakmakci et al., 2006). At one end of the spectrum is virtual reality (VR)
display, which effectively extends the field of view (FOV), blocks the entire ambient, and offers an
immersive virtual environment independent of the user’s real surroundings. At the other end of the
spectrum is augmented reality (AR) display, which not only pursues high-quality see-through
performance but also enriches the real world by overlaying digital contents. With advanced level of
optical technology and refreshing user experience, AR and VR displays exhibit potential to trigger
attractive applications, including but not limited to healthcare, education, engineering design,
manufacturing, retail, and entertainment.
The ideal goal of AR and VR display development is to offer reality-like crystal-clear images that can
simulate, merge into, or rebuild the surrounding environment and avoid wearing discomfort concurrently.
This is still challenging at the present stage, especially for AR systems, since most components demand
not only further performance enhancement but also miniaturization in both form factor and power
consumption.
In this paper, we share a few perspectives about the development of optical technologies for AR and VR
head-mounted displays. We begin the discussion by reviewing the visual requirement poised by the

human visual systems. Next, we discuss how emerging optical technologies can help meet these
challenges in terms of resolution, visual comfort, FOV, and dynamic range. Moreover, form factor and
power efficiency are also taken into consideration because they play crucial roles in near-eye display
designs, especially for consumer applications.
REQUIREMENT OF HUMAN VISUAL SYSTEM
To better understand the goal and underlying challenges, it is necessary to examine the performance
parameters of human visual system. The FOV has the distribution plotted in Figure 1A. The monocular
FOV of human eye is about 160° (horizontal) by 130° (vertical). The combined binocular FOV is about
200° (horizontal) by 130° (vertical), with an overlapped region of 120° horizontally (Wheelwright et al.,
2018). The resolution limit of human eye is determined by the average spacing of cone cells in the fovea.
This estimation yields the visual angle of about 0.5 arcmins (Curcio et al., 1990), or 120 pixel-per-degree
(ppd), which corresponds to 20/10 visual acuity. As it comes to display design, there is an apparent trade-
off between resolution density and FOV, given that the total number of display pixels is fixed.
For VR, a broad FOV that covers the human visual range is relatively easy to achieve by designing an
eyepiece with sufficiently low f/#. The main issue becomes the resultant low-resolution density, which
brings up the so-called screen-door effect that considerably compromises the viewing experience. A direct
solution, of course, is to increase the display resolution, which is unfortunately very challenging
considering the high cost and data transport rate. For estimation, to achieve a monocular vision with 100°
FOV and resolution density of 60 ppd (1 arcmin, or 20/20 vision), a display with 6K resolution in
horizontal is required. Some commercial products (like Pimax Vision 8K) now can provide about 4K
monocular resolution, but the daunting price that comes with the high performance remains an issue.
Another approach considers the fact that the high-resolution density only exists within the fovea region of
±2.5° (Rossi and Roorda, 2010), out of which the visual acuity drops drastically (Figure 1B). Therefore,
the high resolution is only required in the central viewing zone, which brings out the concept of foveated
display (Tan et al., 2018; Kim et al., 2019). In foveated displays, the resolution is variant across the entire
viewing region, usually through an optical combination of two display panels that individually address
central and peripheral areas. This way, not only the burden of display hardware is lessened, the
computational and data-transferring burdens are also reduced significantly.
Regarding AR systems, although the trade-off between FOV and resolution density still exists, a more
significant concern is to produce a decent FOV in the first place. Throughout various optical architectures
from free-space combiners, total internal reflection (TIR) freeform combiners (Hua, et al., 2013) to
lightguide combiners, the maximum achievable FOV typically does not exceed 60° in horizontal, which
still has a long path to go towards the human vision limit.
Furthermore, as a high-dynamic-range imaging system, the human eye can adapt to a broad range of
illuminance from 10
4
lux of daylight to 10
-3
lux at night (Hoefflinger, 2007). Thus, contrast ratio (CR) is a
critical display parameter. In VR, the issue of contrast is not significant because the influence of
environment light can be neglected. If the stray light inside the headset can be well managed and
suppressed, then CR can reach over 1000:1. In AR, however, due to the high surrounding illuminance, the
requirement for display brightness can be very high. In this case, a more representative parameter to
consider is ambient contrast ratio (ACR), defined as (Lee et al., 2019):
on ambient
off ambient
L L T
ACR
L L T
+ ⋅
=
+ ⋅
(1)
where L
on
(L
off
) represents the display luminance of on- (off-) state, and T is display transmittance. For a
simple estimation, if we assume a display transmittance of 80% and ambient illuminance of 10
4
lux with
Lambertian distribution, an ACR of 2:1 that barely prevents image washout already requires 2500 nits of

display brightness. A better CR of 5:1 for adequate readability even requires 10,000 nits of brightness.
Current AR systems, for comparison, generally can support brightness only up to 500 nits (Lee et al.,
2019), which can only accommodate indoor use (500 lux).
When evaluating the VR/AR systems capable of 3D image generation, yet another aspect to consider for
human vision is the stereo sensation. The natural viewing experience of a 3D object induces vergence cue
(relative rotation of eyes) and accommodation cue (the focus of eyes), which coincide with each other
(Figure 1C). However, in most of current VR systems, a fixed display plane with different rendered
contents for each eye is adopted. The eye accommodation is fixed on the plane and therefore mismatches
with vergence cue, which causes visual fatigue and discomfort, sabotages stereo acuity, and distorts
perceived depth (Hoffman et al., 2008, Watt et al., 2005). This phenomenon is often called vergence-
accommodation conflict (VAC).
VR: RESOLUTION
The current angular resolution of VR displays still falls short of normal 20/20 vision acuity. Most VR
headsets are using one display panel and viewing optics for each eye to provide the stereoscopy effect;
such an old technology can trace back to the nineteenth century (Wheatstone, 1838). The VR optical
layout is essentially an unsophisticated imaging system using the viewing optics to magnify the display
panel. Therefore, from the system perspective, clearer and sharper imagery can be offered by further
improving both display panels and magnifying lenses. The display industry has been pursuing display
panels with higher resolution, power efficiency, dynamic range, and faster response time yet lower cost.
The fast-evolving flat panel display in the past decade is one of the cornerstones of current VR headsets,
and their future development will also considerably benefit the VR industry. It is vital to increase the
pixel number and density on physical display panels and thus reduce the screen-door effect in the long
term. However, this may bring a heavy burden on image rendering, driving circuits, and power
consumption.
In the meantime, some emerging approaches can offer decent visual experience based on the off-the-shelf
display panels (Figure 2). For global resolution enhancement, the conventional wobulation method (Allen
and Ulichney, 2005) designed for projection displays can be extended to VR. Lee et al., 2017
demonstrated an optical wobulation VR system by synchronizing a switchable liquid crystal
Pancharatnam-Berry phase deflector and subframe images, increasing the pixel density through time-
multiplexing. Zhan et al., 2019 further advanced this approach using a passive polymer deflector and a
polarization management layer, doubling the apparent pixel density without reducing the original frame
rate. More recently, Neguyen et al., 2020 realized mechanical wobulation for both micro-OLED and LCD
panels to reduce the screen-door effect. These prior arts, based on the wobulation method, can simulate
high resolution imagery for the entire FOV before ideal display panels are available. Nonetheless, the
wobulation method still requires a large amount of data rate and cannot reduce the burden placed by the
massive amount of data flow.
Alternatively, the foveation approach aimed at local resolution enhancement can avoid this problem,
which makes use of the non-uniform angular resolution distribution of the human visual system (Rossi
and Roorda, 2010). It offers high resolution on the fovea region of eye retina while maintaining degraded
resolution on the peripherals. This principle was adopted for imaging before near-eye displays (Hua and
Liu, 2007). Generally, in most foveated VR systems, a beam splitter is employed to combine the images
displayed on the low-resolution panel and high-resolution one, resulting in a larger device volume.
Miniaturizing the optical layout and finding an alternative to the bulky beam splitter design is an essential
task for the future development of foveated VR devices. A promising candidate is using an off-axis mini-
projection unit together with a transparent projection screen on top of the display panel. The projection
screen should be transparent for the display light but manifest strong scattering for the off-axis projection
light. A decent example of such a projection screen is polymer-dispersed liquid crystal film with

customized molecular orientation and index mismatch (He et al., 2020). Moreover, since the gaze point is
not always fixed at the center FOV, another potential development direction for the foveation method is
image shifting, which is similar to but more complicated than beam steering technologies. Both
mechanical and optical shitting method for VR displays have been demonstrated, using a rotatable beam
splitter (Sahlsten, 2020) and a switchable liquid crystal deflector (Tan et al. 2018), respectively.
VR: VIEWING OPTICS
In parallel, a decent optical imaging part is also critical for generating high-resolution virtual images in
VR headsets. Due to ergonomic requirements, the viewing optics should be compact and lightweight,
which brings a significant sacrifice in imaging quality. Conventional aspheric singlet with smooth
surfaces usually have limited stray light but a large volume and weight. Thus, its compact Fresnel
alternative is more prevalent in current commercial VR headsets (Geng et al., 2018). Although Fresnel
singlets have more degrees of freedom for aberration control, its intrinsic diffractive artifacts and
unavoidable stray light considerably reduce the image sharpness. For now, the systematic imaging quality
is limited by the display panel resolution in most headsets, so these drawbacks of Fresnel lenses are still
tolerable. But in the long run, these issues could become more critical as display pixel density gradually
increases. To further reduce the device dimension, catadioptric pancake optics can be employed (Wong et
al., 2017). With reflective surfaces induced to share the optical power of refractive components, the
pancake lenses can allow display panel with smaller sizes due to their shorter focal length. However,
these benefits come at the cost of 75% light efficiency and demanding polarization control to eliminate
ghost images. In this case, plastic materials with limited birefringence and high-quality polarizers and
waveplates are highly demanded.
Moreover, the emerging flat optics including broadband diffractive lenses (Meem et al., 2020),
metalenses (Chen et al., 2019), and liquid crystal Pancharatnam-Berry phase lenses (Zhan et al., 2019)
can also be applied in the VR lens system for aberration control and system miniaturization. By adding a
thin-film flat polymer lens, it is possible to sharpen the imagery by more than three times (Zhan et al.,
2020). Another intriguing approach is to use a two-dimensional curved display (Grover et al., 2018). With
the field curvature compensated by the tailored panel curvature, the heavy burden on the lens design can
be well relieved. Alternatively, the curved fiber faceplate (Zhao et al., 2019) can be attached to the
display panel as a surface-shaping component, which can be designed together with the viewing optics for
sharper imaging.
VERGENCE-ACCOMMODATION CONFLICT
Aside from limited resolution and screen-door effect, VAC is another significant issue in VR systems. A
plethora of solutions have been developed to mitigate this conflict (Kramida, 2015), but only few have
been applied to the current commercial VR headsets. Monovision displays represent a simple solution to
VAC, where vergence is not present for the virtual image. Since only one eye is offered with digital
images, this approach is more suitable for specific AR applications but not immersive VR. The other
extreme is accommodation-invariant approaches, like the Maxwellian view (Takaki and Fujimoto, 2018),
where the point source is focused on the pupil with angularly encoded amplitude information, and the
image on the retina is independent of the accommodation response. However, to tolerate the eye
movement, Maxwellian-view systems usually exhibit a limited FOV.
In general, most of other approaches offer a proper accommodation cue to mimic the retina blur and
therefore alleviate the conflict. A typical example is holographic display (Yamaguchi et al., 2007) aimed
at reconstructing accurate wavefront of the entire 3D scene and offering accurate retinal blur. Aside from
the limited FOV, holographic displays usually manifest degraded image quality due to laser speckles.
Similarly, light field displays (Wetzstein et al., 2012) reconstruct the geometric light rays instead of the

diffractive wavefront, which can also provide the approximately correct depth information and retina blur
but usually end up with a low resolution. If the amount of information is taken into consideration, it is not
surprising that these approaches aimed at showing volumetric information like holograms and light fields
cannot offer sufficient resolution with the limited bandwidth of current hardware. Even so, there is no
denying that these approaches may gradually mature in the long term with better hardware and eventually
become satisfactory for users.
In the short term, methods that can find an acceptable trade-off between depth accuracy and system
complexity should be more practical for addressing the VAC in current commercial products, such as
varifocal and multifocal displays. Varifocal displays employ an eye tracker to locate the gaze location and
an adaptive focusing component to shift the display depth accordingly. Additionally, real-time blur
rendering is also preferred in varifocal approaches because they cannot naturally generate retina blur
(Dunn et al., 2017). In comparison, multifocal displays (Liu and Hua, 2010; Hua, 2017; Zhan et al., 2018;
Tan et al., 2018; Liu et al., 2018) can create near-correct physical depth blur and offer a customizable
balance between depth accuracy and hardware bandwidth by choosing the density of focal planes for
different applications. A systematic summary and analysis of multifocal displays can be found in Zhan et
al., 2020. For both varifocal and multifocal displays, the need for high-quality focal changing components
is still urgent, which should have fast response time, compact form factor, and low power consumption.
AR: FIELD OF VIEW
Different from the immersive experience provided by VR, one of the most pressing challenges in AR is
expanding the FOV. Due to various designs and form factors for the same type AR, we will discuss and
compare the diagonal FOV instead of the horizontal/vertical FOV values. The diagonal FOV is related to
the horizontal/vertical FOV as 2 2
diagonal horizontal vertical
FOV FOV FOV= +
. To address the inadequate FOV issue,
we will overview potential solutions and analyze the systems case by case. In a lightguide-based near-eye
display (LNED), the light from optical engine propagates inside the lightguide following the TIR and is
then extracted to human eye by an exit pupil expansion (out-coupler) as illustrated in Figure 3A.
Typically, the core optical elements in such a system are the image source and the light combiner
consisting of an input coupler and an output coupler. The optical engine can be a liquid- crystal-on-silicon
(LCoS) panel, digital light processing (DLP), µOLED, µLED, and laser beam scanning (LBS) (Kress,
2020), while the combiners can be a reflective mirror or diffractive grating (Kress, 2019; Lee et al., 2019).
When the light propagating inside the lightguide, the TIR angle is governed by the refractive index of the
lightguide. Meanwhile, the index contrast of the coupler determines the angular and spectral responses,
especially for grating and hologram, which affects the color uniformity over the FOV and the eye-box
(Kress, 2019). Due to the significant impact of the coupler on the system, numerous technologies have
been applied to optimize the coupler performance (Xiang et al., 2018; Gao et al., 2017; Yin et al., 2019;
Yin et al., 2020). As a result, the angular response of a LNED system is not limited by the coupler but by
the critical angle of TIR, which is in turn determined by the lightguide refractive index. The normal
refractive index of lightguide is n
d
=1.50±0.03 (Sprengard et al., 2019), while a comparatively high
refractive index is n
d
=1.7−1.8 (Masuno et al., 2019). For most LNEDs, such as HoloLens 2 and Magic
Leap One, high index glass has been implemented to realizing a diagonal FOV of 50° (Kress, 2020). To
widen FOV further, a high index n
d
≥1.9 glass has been commercialized recently. By using such a high-
index glass, the critical angle becomes smaller so that the range from critical angle to 90° gets larger,
meaning a wider FOV can be supported in the lightguide.
In addition to improving the intrinsic characteristics of the components, such as increasing the refractive
index of glass or widening the angular bandwidth of coupler, the FOV can also be extended by expanding
the system’s degree of freedom. Through utilizing the multiplexing of coupler functions, such as spatial
multiplexing (Vallius et al., 2017), polarization multiplexing (Shi et al., 2018), etc., we can build a more

sophisticated system with wide FOV. The multiplexing method utilized for broadening FOV is essentially
to stitch images based on different characteristics of light, thereby realizing a more informative and
realistic experience. However, it is worth mentioning that the multiplexing is not limited in benefitting the
FOV, it also plays an essential role in overcoming the VAC issue (Zhan et al., 2019; He et al., 2020) and
presenting full-color images (Jang et al., 2017) in the AR system.
In a near-eye display, the multiplexing based on the properties of light can be categorized into spatial
multiplexing, time multiplexing, polarization multiplexing, wavelength multiplexing, and angular
multiplexing. Sometimes, more than one method is used in a system. By spatially combining two images
to increase the FOV, Microsoft patented a combiner structure with two intermediate couplers separated
spatially (Vallius et al., 2017). Then Shi et al. proposed the polarization multiplexing based on meta-
gratings (Shi et al., 2018). Similar to polarization division multiplexing in optical fiber communications
where two channels with orthogonal polarizations are used to double the information capacity, the
polarization multiplexing method increases the FOV by encoding the left and right FOVs into two
orthogonal polarization channels, TE and TM, respectively. Recently, Yoo et al. propose an extended
FOV LNED system by polarization multiplexing using LC-based grating (Yoo et al., 2020). In the
holographic volume grating (HVG)-based LNEDs, several multiplexing techniques have been reported.
Han et al. (2015) and Yu et al. (2017) attempted to apply the spatially multiplexing in out-coupler HVG to
obtain wide FOV. Lately, LC-based polarization volume gratings (PVGs), also known as Bragg
polarization gratings, with high diffraction efficiency and large angular bandwidth have been reported
(Lee et al., 2017; Yin et al., 2020). Due to these special optical features, it is feasible to build a spatially
multiplexed AR system with a large FOV using PVGs. As depicted in Figure 3B, the image information
is coupled into two lightguides through two input couplers that are spatially separated. Then the light
propagates into the output area through TIR, and the image information is extracted by two output
couplers with different periodicity and form a larger FOV beyond the limitation of lightguide TIR. Since
the asymmetric input and output coupler here may induce significant chromatic aberrations and image
distortion, it is preferable to employ narrow-band display engine and anamorphic image pre-processing.
The Maxwellian view is an observation method, in which the lens system forms an image of the light
source in the plane of the observer’s pupil, instead of looking at the source directly. Therefore, the effect
of the eye’s optical aberrations is minimized, and the quantity of light independent of pupil size is
increased (Westheimer, 1966; Sugawara et al., 2016). When applying this method in NEDs, the effective
eye pupil can be regarded as a tiny aperture, and the focal depth of the image will be dramatically
increased. Therefore, the system offers focus-free feature, i.e., no matter where the eye focuses, the image
is always clear. However, this method has its own limitations, especially the severely reduced eyebox. To
address this issue, Kim et al. (Kim et al., 2018) combined a Maxwellian view LNED with holographic
optical element (HOE) multiplexing to obtain an enlarged eyebox or a steering eyebox.
Figure 3C illustrates a typical schematic diagram of the Maxwellian view system. Based on geometric
optics, the Maxwellian view system can evolve into different forms, such as partially reflective elements
and LNEDs. From Figure 3C, the FOV of this system is directly related to the numerical aperture (NA) of
the lens system. With rapid technology development and urgent needs from industry, numerous novel flat
lenses with a wide acceptance angle and large aperture in both on-axis and off-axis types have emerged
(Khorasaninejad et al., 2016; Yin et al., 2020). Based on the HOE with a large NA, NVIDIA
demonstrated an 85° × 78° monocular FOV Maxwellian view system (Kim et al., 2019). Further efforts
have been investigated to enlarge the FOV. Xiong et al. 2020 demonstrated a large FOV AR system with
100° diagonal FOV by hybridizing the Maxwellian view and the lightguide-based exit pupil expander. By
increasing the NA and compressing the lens volume, both FOV and form factor of the Maxwellian-view
based NED system can be improved significantly.

AR: BRIGHTNESS AND EFFICIENCY
For optical see-through AR displays, ACR is a critical parameter, which puts a strict requirement on
display luminance (Lee et al., 2019). As a general guideline, for indoor applications, the output luminance
of the AR display should be at least 500 nits. By contrast, for outdoor applications, the required
luminance would exceed 10,000 nits. To deliver such a high luminance, both microdisplay and efficient
relay/combiner optics are pivotal.
A roadmap of potential display engines is plotted in Figure 4. To provide a more general guideline on
how to choose display engines, a qualitative comparison among five candidates is summarized in Table 1.
Field-sequential LCoS is a reflective display based on polarization modulation of backlight (Huang et al.,
2018). Due to high brightness (10
4
to 10
5
nits) and commercial availability, it has been used in Magic
Leap One (Klug et al., 2016) and HoloLens (Kress SID 2017). A proper polarization conversion system
(PCS) can boost the efficiency and brightness of an LCoS since only light with a certain linear
polarization can be reflected by the polarization beam splitter (PBS) and modulated by the LCoS. In
traditional, large size LCoS projectors, a PCS consisting of a fly-eye lens, a PBS array, and a patterned
half-wave plate is integrated. However, as the form factor shrinks to microdisplay sizes, fabrication
difficulties and bulkiness of such a PCS has its limitation. Although some researchers proposed improved
PCSs based on thin-film polarization gratings (Kim et al., 2012; Du et al., 2015), the small form factor,
large angular bandwidth, and high efficiency are still lacking. Another fundamental issue of LCoS is its
limited dynamic range, as the relatively poor dark state will influence the see-through experience,
especially for indoor uses. A two-dimensional (2D) illumination or backlight with independently
addressable patches offers a promising solution, like the mini-LED array for LCD panels (Tan et al.,
2018). Similar to LCoS, DLP panels are field-sequential micromirror displays with high brightness
(Thompson et al., 2015), as employed by DigiLens. Compared to LCoS, the amplitude modulation of
DLP is polarization independent, and the dynamic range can be higher. For both reflective microdisplay
panels (LCoS and DLP), while LEDs are typically applied as the illumination source, other light sources,
such as lasers, are also available. Lasers are inherently collimated and linearly polarized and are very
suitable for LCoS. However, additional de-speckle optics are needed in order to achieve good image
quality.
In comparison with projection, emissive displays are less mature but have potential to reduce the form
factor. They exhibit intrinsically high dynamic range because of the true black state. Micro organic light-
emitting diode (μOLED) is a promising candidate for emissive microdisplays. The typical architecture is
patterned color filters on top of white OLEDs. To date, full-color μOLED displays with 3,000 to 5,000
nits in luminance and ~3,000 ppi (pixel per inch) in resolution have been achieved (Haas, 2018;
Motoyama et al., 2019). But for AR displays with a large eye-box, such brightness is still inadequate (Lee
et al., 2019). Future development should pay attention to boosting their brightness, device lifetime, and
current efficiency. On the other hand, micro light-emitting diode (μLED) is emerging and has potential to
become the next-generation display technology. The most recent development of 10-μm pitch (~1300 ppi)
full-color LED microdisplay has achieved 10
5
to 10
6
nits in luminance (Quesnel et al., 2020). Despite this
impressive progress, μLED still faces two major challenges. The first is to enhance the non-radiative
recombination when the area ratio of the side wall increases (Gou et al., 2019). This means, for small
μLED chips down to <5 μm, the external quantum efficiency would drop dramatically. The second issue
is how to realize full color and high resolution simultaneously, as mass transfer and assembly for such
tiny RGB LEDs is challenging (Lin et al., 2020; Wong et al., 2020). A parallel approach is to use blue
μLED to pump green and red quantum dots as color conversion (Huang et al., 2020). However, obtaining
a uniform, long lifetime color conversion layer without color crosstalk for such small pixel sizes is by no
means easy. Therefore, further effort is needed to develop mass transfer technique or color conversion
layer patterning technique for ultra-small pixel pitch (<5 µm) μLEDs.
As for scanning display systems, they are normally with high efficiency, small form factor, high dynamic
range, and high brightness using laser illumination. Typically, a 2D micro-electromechanical system

(MEMS) mirror or two 1D MEMS mirrors are applied to scan the laser beam in orthogonal directions to
form 2D images. Different from the panel-based displays, scanning displays do not have an object plane.
This unique property indicates that unlike panel-based displays form object images on the panel, the
scanning displays can directly form images on the retina. One prominent example is the laser beam
scanning system in North Focals (Alexander et al., 2018). As most scanning display engines have
intrinsically small exit pupil, they need a proper exit pupil expansion/steering, and thus the optical design
will be more sophisticated. In comparison with reflective and emissive displays, the image uniformity of
the scanning method is another inevitable issue that requires improvement.
The information generated from the optical engine will undergo magnifying optics and/or combiners and
finally project into human eyes. The combiners can be classified into two types: reflective and diffractive.
The reflective type includes freeform half mirrors, freeform prisms, birdbath combiners, and cascaded
mirrors (Wei et al., 2018; Cheng et al., 2011), while diffractive type covers all kinds of grating-coupler
based lightguide combiners and off-axis holographic optical element (HOE, not used in lightguide)
combiners (Li et al., 2016). Their schematic plots are shown in Figure 4, and a comparison among them is
illustrated in Table 2.
The freeform half mirrors, freeform prisms and birdbath combiners usually manifest decent imaging
quality and high optical efficiency, but mainly suffer from a large form factor. To reduce the form factor,
cascaded mirrors embedded in a lightguide has been invented. However, for lightguide combiners,
additional attention should be paid on see-through transmittance, see-through uniformity, stray light
control and image brightness uniformity. As a result, the image quality and optical efficiency are usually
compromised. The diffractive combiners are also introduced to reduce the form factor of traditional
reflective combiners. Different from the reflective counterpart, the chromatic nature of diffractive
elements needs to be considered in optical design. Off-axis HOEs combined with an LBS system can
provide a true glasses-like form factor yet a limited eye-box. To further enlarge the eye-box, grating-
coupled lightguide combiners are employed where the output coupler design is more complicated since it
can also perform as the exit pupil expander.
Currently, two types of gratings are employed in lightguide AR: holographic volume Bragg gratings
(VBGs) and surface relief gratings (SRGs). Due to the different refractive index contrast, they exhibit
different spectral and angular responses. The traditional VBGs with a small refractive index contrast
(δn≤0.05) manifest narrow spectral (~10 nm) and angular (~5° in air) bandwidths, while SRGs with a
large δn (≥0.5) show much broader spectral and angular bands (Lee et al., 2019). Interestingly, DigiLens
has developed a large δn VBG (close to LC birefringence) based on holographic polymer-dispersed liquid
crystal (HPDLC), which is switchable and it performs much better than traditional VBG (Brown et al.,
2018). Beside these two gratings, polarization volume gratings (PVGs) based on chiral liquid crystals
(CLCs) are also emerging (Yin et al., 2019). The refractive index contrast is essentially the birefringence
of the LC material and thus can be tuned within a broad range (from <0.1 to >0.4). As those grating
couplers are usually optimized for a particular polarization (e.g., a linear polarization for VBGs and SRGs,
and a circular polarization for PVGs), a PCS modulating the polarization of light from the display engine
and polarization management within the lightguide will be significant for improving the system efficiency.
Another unavoidable aspect of improving light efficiency is the 2D exit pupil expander (EPE) design.
Typically, a turn-around gradient-efficiency grating (also termed as fold grating) is performed to first
expand the eye-box in one direction within the lightguide. Then the output grating extends the eye-box in
another direction. Specifically, due to the inherent chromatic dispersion in diffraction, color uniformity
control is as challenging as brightness uniformity in most of the waveguide designs using diffractive
combiners. However, since there is a trade-off between optical efficiency of the gratings (both the turn-
around grating and the output grating) and color/brightness uniformity within the expanded eye-box,
finding an appropriate balance between them is essential from the system perspective.

Conclusion
In conclusion, we overviewed the major challenges and discussed potential opportunities of display optics
in the fast-developing field of AR and VR systems. The requirements from the human visual system are
analyzed in detail to offer quantitative standards for future near-eye display devices. These requirements
also bring out the major issues that need to be emphasized and addressed in current devices, regarding
panel resolution, form factor, imaging performance, VAC, FOV, and brightness. By learning from recent
advances in optics and developing trends of AR and VR devices, we shared a few thoughts about how to
meet these challenges in the near future and the long run.
Acknowledgments
This work was supported by Intel Corporation and GoerTek Electronics.
Author Contributions
Conceptualization, T.Z. and K.Y.; Methodology, T.Z. and K.Y.; Writing – Original Draft T.Z. K.Y. J.X.
and Z.H.; Writing – Review & Editing T.Z. K.Y. J.X. Z.H. and S.T.W; Supervision, S.T.W.
Declaration of Interests
The authors declare no competing interests.
References
Alexander, S., Bailey, M., Morrison, V.R., Holland, L.F., and Moore, J. (2018). Systems, devices, and
methods for eyebox expansion in wearable heads-up displays. U.S. Patent No. 9,989,764.
Allen, W. and Ulichney, R. (2005, May). 47.4: Invited paper: Wobulation: Doubling the addressed
resolution of projection displays. In SID Symposium Digest of Technical Papers (Vol. 36, No. 1, pp.
1514-1517). Oxford, UK: Blackwell Publishing Ltd.
Brown, R.D., Grant, A.J., Hendrick, W.L., Popovich, M.M., Stanley, J.H., and Waldern, J.D. (2018).
Transparent Waveguide Display. U.S. Patent Application No. 15/943,590.
Cakmakci, O. and Rolland, J. (2006). Head-worn displays: a review. J. Disp. Technol. 2, 199-216.
Chen, H., Lee, J. H., Lin, B.Y., Chen, S., and Wu, S.T. (2018). Liquid crystal display and organic light-
emitting diode display: present status and future perspectives. Light: Sci. & Appl. 7, 17168.
Chen, W.T., Zhu, A.Y., Sisler, J., Bharwani, Z. and Capasso, F. (2019). A broadband achromatic
polarization-insensitive metalens consisting of anisotropic nanostructures. Nature Communications 10,
355.
Cheng, D., Wang, Y., Hua, H., and Sasian, J. (2011). Design of a wide-angle, lightweight head-mounted
display using free-form optics tiling. Opt. Lett. 36, 2098-2100.
Curcio, C.A., Sloan, K.R., Kalina, R.E., and Hendrickson, A.E. (1990). Human photoreceptor topography.
J. Comparative Neurol. 292, 497-523.
Du, T., Fan, F., Tam, A.M.W., Sun, J., Chigrinov, V.G., and Kwok, H.S. (2015). Complex nanoscale‐
ordered liquid crystal polymer film for high transmittance holographic polarizer. Adv. Mater. 27, 7191-
7195.

Dunn, D., Tippets, C., Torell, K., Kellnhofer, P., Akşit, K., Didyk, P., Myszkowski, K., Luebke, D. and
Fuchs, H. (2017). Wide field of view varifocal near-eye display using see-through deformable membrane
mirrors. IEEE transactions on visualization and computer graphics 23, 1322-1331.
Gao, K., McGinty, C., Payson, H., Berry, S., Vornehm, J., Finnemeyer, V., Roberts, B. and Bos, P. (2017).
High-efficiency large-angle Pancharatnam phase deflector based on dual-twist design. Opt. Express 25,
6283-6293.
Geng, Y., Gollier, J., Wheelwright, B., Peng, F., Sulai, Y., Lewis, B., Chan, N., Lam, W.S.T., Fix, A.,
Lanman, D. and Fu, Y. (2018, May). Viewing optics for immersive near-eye displays: pupil swim/size
and weight/stray light. In Digital Optics for Immersive Displays (Vol. 10676, p. 1067606). International
Society for Optics and Photonics.
Gou, F., Hsiang, E.L., Tan, G., Chou, P. T., Li, Y. L. Lan, Y. F., and Wu, S. T. (2019). Angular color
shift of micro-LED displays. Opt. Express 27, A746-A756.
Grover, G., Salahieh, B. and Nestares, O. (2018, June). Curved screen virtual reality headsets. In 3D
Image Acquisition and Display: Technology, Perception and Applications (pp. 3M5G-3). Optical Society
of America.
Haas, G. (2018). Microdisplays for Augmented and Virtual Reality. SID Symp. Dig. Tech. Papers 49,
506-509.
Han, J., Liu, J., Yao, X., and Wang, Y. (2015). Portable waveguide display system with a large field of
view by integrating freeform elements and volume holograms. Opt. Express 23, 3534-3549.
He, Z., Yin, K., and Wu, S.T. (2020). Passive polymer-dispersed liquid crystal enabled multi-focal plane
displays. Opt. Express 28, 15294-15299.
Hoefflinger, B. (2007). High-dynamic-range (HDR) vision. Springer Berlin Heidelberg.
Hoffman, D. M., Girshick, A. R., Akeley, K., and Banks, M. S. (2008). Vergence-accommodation
conflicts hinder visual performance and cause visual fatigue. J. Vision 8, 1–30.
Hua, H. (2017). Enabling focus cues in head-mounted displays. Proceedings of the IEEE 105, 805-824.
Hua, H. and Liu, S. (2008). Dual-sensor foveated imaging system. Appl. Opt. 47, 317-327.
Hua, H., Hu, X. and Gao, C. (2013). A high-resolution optical see-through head-mounted display with
eyetracking capability. Opt. Express 21, 30993-30998.
Huang, Y., Liao, E., Chen, R., and Wu, S.T. (2018). Liquid-crystal-on-silicon for augmented reality
displays. Appl. Sci. 8, 2366.
Huang, Y., Hsiang, E.L., Deng, M.Y., and Wu, S.T. (2020). Mini-LED, Micro-LED and OLED displays:
Present status and future perspectives. Light: Sci. & Appl. 9, 105.
Jang, C., Bang, K., Kim, J., Jeong, Y., and Lee, B. (2017). Full color virtual retinal display using a
holographic optical element. Imaging and Applied Optics (OSA) paper JTu5A.32.
Khorasaninejad, M., Chen, W. T., Devlin, R. C., Oh, J., Zhu, A. Y., and Capasso, F. (2016). Metalenses at
visible wavelengths: Diffraction-limited focusing and subwavelength resolution imaging. Science 352,
1190-1194.
Kim, J., Jeong, Y., Stengel, M., Akşit, K., Albert, R., Boudaoud, B, Greer, T., Kim, J., Lopes, W.,
Majercik, Z., et al. (2019). Foveated AR: dynamically-foveated augmented reality display. ACM
Transactions on Graphics (TOG) 38, 1-15.

Kim, J., Jeong, Y., Stengel, M., Akşit, K., Albert, R., Boudaoud, B., Greer, T., Kim, J., Lopes, W.,
Majercik, Z. and Shirley, P. (2019). Foveated AR: dynamically-foveated augmented reality display. ACM
Trans. Graph. 38, 1-15.
Kim, J., Komanduri, R.K., Lawler, K.F., Kekas, D.J., and Escuti, M.J. (2012). Efficient and monolithic
polarization conversion system based on a polarization grating. Appl. Opt. 51, 4852-4857.
Kim, S.B. and Park, J.H. (2018). Optical see-through Maxwellian near-to-eye display with an enlarged
eyebox. Opt. Lett. 43, 767-770.
Klug, M.A., Cahall, S.C., and Chung, H. (2016). Separated pupil optical systems for virtual and
augmented reality and methods for displaying images using same. U.S. Patent Application No.
15/146,296.
Kramida, G. (2015). Resolving the vergence-accommodation conflict in head-mounted displays. IEEE
Transactions on Visualization and Computer Graphics 22, 1912-1931.
Kress, B.C. (2019, July). Optical waveguide combiners for AR headsets: features and limitations. In
Digital Optical Technologies 2019 (Vol. 11062, p. 110620J). International Society for Optics and
Photonics.
Kress, B.C. (2020). Optical Architectures for Augmented-, Virtual-, and Mixed-Reality Headsets.
Kress, B.C. and Cummings, W.J. (2017, May). 11‐1: Invited paper: Towards the ultimate mixed reality
experience: HoloLens display architecture choices. In SID symposium digest of technical papers (Vol. 48,
No. 1, pp. 127-131).
Lee, Y.H., He, Z., and Wu, S.T., (2019). Optical properties of reflective liquid crystal polarization volume
gratings. J. Opt. Soc. Am. B 36, D9-D12.
Lee, Y.H., Yin, K., and Wu, S.T. (2017). Reflective polarization volume gratings for high efficiency
waveguide-coupling augmented reality displays. Opt. Express 25, 27008-27014.
Lee, Y.H., Zhan, T., and Wu, S.T. (2017). Enhancing the resolution of a near-eye display with a
Pancharatnam–Berry phase deflector. Opt. Lett. 42, 4732-4735.
Lee, Y.H., Zhan, T., and Wu, S.T. (2019). Prospects and challenges in augmented reality displays. Virtual
Reality & Intelligent Hardware 1, 10-20.
Li, G., Lee, D., Jeong, Y., Cho, J., and Lee, B. (2016). Holographic display for see-through augmented
reality using mirror-lens holographic optical element. Opt. Lett. 41, 2486-2489.
Lin, J.Y. and Jiang, H.X. (2020). Development of microLED. Appl. Phys. Lett. 116, 100502.
Liu, S. and Hua, H. (2010). A systematic method for designing depth-fused multi-focal plane three-
dimensional displays. Opt. Express 18, 11562-11573.
Liu, S., Li, Y., Zhou, P., Chen, Q., Li, S., Liu, Y., Wang, Y. and Su, Y. (2018). Full‐color multi‐plane
optical see‐through head‐mounted display for augmented reality applications. J. Soc. Inf. Disp. 26, 687-
693.
Masuno, A., Iwata, T., Yanaba, Y., Sasaki, S., Inoue, H., and Watanabe, Y. (2019). High refractive index
La-rich lanthanum borate glasses composed of isolated BO
3
units. Dalton Trans. 48, 10804-10811.
Meem, M., Banerji, S., Pies, C., Oberbiermann, T., Majumder, A., Sensale-Rodriguez, B. and Menon, R.
(2020). Large-area, high-numerical-aperture multi-level diffractive lens via inverse design. Optica 7, 252-
253.
Motoyama, Y., Sugiyama, K., Tanaka, H., Tsuchioka, H., Matsusaki, K., and Fukumoto, H. (2019).
High efficiency OLED microdisplay with microlens array. J. Soc. Inf. Disp. ‐27, 354-360.

Nguyen, J., Smith, C., Magoz, Z. and Sears, J. (2020, February). Screen door effect reduction using
mechanical shifting for virtual reality displays. In Optical Architectures for Displays and Sensing in
Augmented, Virtual, and Mixed Reality (AR, VR, MR) (Vol. 11310, p. 113100P). International Society
for Optics and Photonics.
Quesnel, E., Lagrange, A., Vigier, M., Consonni, M., Tournaire, M., Le Marchand, V., Suhm, A., Demars,
P., Pillet, J.-C., Ben Bakir, B., Olivier, N., Feltin, E., Lamy, J.M., D’Amico, M., Cao, E., Haas, G.,
Charrier, L., and Coni, P. (2020). Dimensioning a full color LED microdisplay for augmented reality
headset in a very bright environment. J. Soc. Inf. Disp. DOI:10.1002/jsid.884.
Rossi, E. A. and Roorda, A. (2010). The relationship between visual resolution and cone spacing in the
human fovea. Nature Neurosci. 13, 156-157.
Sahlsten, O. (2020, April). Immersive Mixed Reality (Conference Presentation). In Optical Architectures
for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) (Vol. 11310, p.
1131024). International Society for Optics and Photonics.
Shi, Z., Chen, W.T., and Capasso, F. (2018, May). Wide field-of-view waveguide displays enabled by
polarization-dependent metagratings. In Digital Optics for Immersive Displays (Vol. 10676, p. 1067615).
International Society for Optics and Photonics.
Sprengard, R., Sjogren, B., Nass, P., Ottermann, C., Lange, B., Damm, T., Plapper, V., Dietrich, V.,
Maurer, U., Fang, H. and Weng, N. (2019). High Refractive Index Glass Wafers for Augmented Reality
Devices using Waveguide Technology: Recent Advances in Control of Quality Parameters and their
Correlation with Device Properties. SID Symp. Dig. Tech. Pap. 50, 116-120.
Sugawara, M., Suzuki, M., and Miyauchi, N. (2016). Retinal Imaging Laser Eyewear with Focus‐Free
and Augmented Reality. SID Symp. Dig. Tech. Pap. 47, 164-167.
Takaki, Y. and Fujimoto, N. (2018). Flexible retinal image formation by holographic Maxwellian-view
display. Opt. Express 26, 22985-22999.
Tan, G., Huang, Y., Li, M.C., Lee, S.L., and Wu, S.T. (2018). High dynamic range liquid crystal displays
with a mini-LED backlight. Opt. Express 26, 16572-16584.
Tan, G., Lee. Y. H., Zhan. T., Yang, J., Liu, S., Zhao, D., and Wu, S. T. (2018). Foveated imaging for
near-eye displays. Opt. Express 26. 25076-25085.
Tan, G., Zhan, T., Lee, Y. H., Xiong, J., and Wu, S.T. (2018). Polarization-multiplexed multi-plane
display. Opt. Lett. 43, 5651-5654.
Thompson, J., Pettitt, G., and Ferri, J. (2015). Practical Application of TI DLP® Technology in the Next
Generation Head-up Display System. SID Symp. Dig. Tech. Papers 46, 700–703.
Vallius, T., and Tervo, J. (2017). U.S. Patent No. 9,791,703. Washington, DC: U.S. Patent and Trademark
Office.
Watt, S. J., Akeley, K., Ernst, M. O., and Banks, M. S. (2005). Focus cues affect perceived depth. J.
Vision 5, 7-7.
Wei, L., Li, Y., Jing, J., Feng, L., and Zhou, J. (2018). Design and fabrication of a compact off-axis see-
through head-mounted display using a freeform surface. Opt. Express 26, 8550-8565.
Westheimer, G. (1966). The maxwellian view. Vision research 6, 669-682.
Wetzstein, G., Lanman, D., Hirsch, M. and Raskar, R. (2012). Tensor displays: compressive light field
synthesis using multilayer displays with directional backlighting. ACM Transactions on Graphics 31, 1-
11.

Wheatstone, C. (1838). XVIII. Contributions to the physiology of vision. Part the first. On some
remarkable, and hitherto unobserved, phenomena of binocular vision. Philosophical transactions of the
Royal Society of London, 128, 371-394.
Wheelwright, B., Sulai, Y., Geng, Y., Luanava, S., Choi, S., Gao, W., & Gollier, J. (2018, May). Field of
view: not just a number. In Digital Optics for Immersive Displays (Vol. 10676, p. 1067604). International
Society for Optics and Photonics.
Wong, M.S., Nakamura, S., and DenBaars, S.P. (2020). Progress in high performance III-Nitride micro-
light-emitting diodes. ECS J. Solid State Sci. Technol. 9, 015012.
Wong, T.L., Yun, Z., Ambur, G. and Etter, J. (2017, June). Folded optics with birefringent reflective
polarizers. In Digital Optical Technologies 2017 (Vol. 10335, p. 103350E). International Society for
Optics and Photonics.
Xiang, X., Kim, J., and Escuti, M.J. (2018). Bragg polarization gratings for wide angular bandwidth and
high efficiency at steep deflection angles. Sci. Rep. 8, 7202.
Xiong, J., Tan, G., Zhan, T. and Wu, S.T. (2020). Wide‐view augmented reality display with diffractive
cholesteric liquid crystal lens array. J. Soc. Inf. Disp. 28, 450-456.
Yamaguchi, T., Okabe, G. and Yoshikawa, H. (2007). Real-time image plane full-color and full-parallax
holographic video display system. Opt. Eng. 46, 125801.
Yin, K., He, Z., and Wu, S.T. (2020). Reflective Polarization Volume Lens with Small f Number and ‐
Large Diffraction Angle. Adv. Opt. Mater. 8, 2000170.
Yin, K., Lin, H.Y., and Wu, S.T. (2019). Chirped polarization volume grating with ultra-wide angular
bandwidth and high efficiency for see-through near-eye displays. Opt. Express 27, 35895-35902.
Yin, K., Lin, H.Y., and Wu, S.T. (2020). Chirped polarization volume grating for wide FOV and high
efficiency waveguide-based AR displays. J. Soc. Inf. Disp. 28, 368-374.
Yoo, C., Bang, K., Chae, M., and Lee, B. (2020). Extended-viewing-angle waveguide near-eye display
with a polarization-dependent steering combiner. Opt. Lett. 45, 2870-2873.
Yu, C., Peng, Y., Zhao, Q., Li, H., and Liu, X. (2017). Highly efficient waveguide display with space-
variant volume holographic gratings. Appl. Opt. 56, 9390-9397.
Zhan, T., Lee, Y.H. and Wu, S.T. (2018). High-resolution additive light field near-eye display by
switchable Pancharatnam–Berry phase lenses. Opt. Express 26, 4863-4872.
Zhan, T., Lee, Y.H., Tan, G., Xiong, J., Yin, K., Gou, F., Zou, J., Zhang, N., Zhao, D., Yang, J. and Liu, S.
(2019). Pancharatnam–Berry optical elements for head-up and near-eye displays. J. Opt. Soc. Am. B 36,
D52-D65.
Zhan, T., Xiong, J., Tan, G., Lee, Y.H., Yang, J., Liu, S. and Wu, S.T. (2019). Improving near-eye
display resolution by polarization multiplexing. Opt. Express 27, 15327-15334.
Zhan, T., Xiong, J., Zou, J. and Wu, S.T. (2020). Multifocal displays: review and prospect. PhotoniX 1,
10.
Zhan, T., Zou, J., Lu, M., Chen, E., and Wu, S. T. (2019). Wavelength-multiplexed multi-focal-plane
seethrough near-eye displays. Opt Express 27, 27507-27513.
Zhan, T., Zou, J., Xiong, J., Liu, X., Chen, H., Yang, J., Liu, S., Dong, Y. and Wu, S.T. (2020). Practical
Chromatic Aberration Correction in Virtual Reality Displays Enabled by Cost‐Effective Ultra‐Broadband
Liquid Crystal Polymer Lenses. Advanced Optical Materials 8, 1901360.

Zhao, R., Zhou, Y., Wei, D., Wang, Y., Fu, Y. and Wang, J. (2019, December). A high-resolution optical-
fiber imaging sensor. In AOPC 2019: Optical Fiber Sensors and Communication (Vol. 11340, p.
1134007). International Society for Optics and Photonics.
Figures
Figure 1. Illustration on the performance of human vision. (A) The profile of human FOV. (B) The
relation between human visual acuity and visual angle. (C) Sketch of the VAC issue. The accommodation
cue coincides with vergence cue when viewing a real object (left). The mismatch occurs when viewing a
virtual object displayed at a fixed plane (right).

Figure 2. The development trend of panel resolution. The pixel density of display panels will gradually
increase for VR application. Before panels with ideal pixel density are available at low cost, it is also
feasible to employ global resolution enhancement based on mechanical or optical wobulation method and
local resolution enhancement with foveated display technologies.

Figure 3. Optical structures of AR systems with extended FOV. (A) Schematic illustration of the
LNED system. TIR happens at each reflection during the propagation, and the angle is marked in orange.
(B) Lightguide-based polarization multiplexing system for enlarging FOV. The system is based on two
PVGs with opposite polarization responses (LCP and RCP) and different diffraction angles. (C)
Schematic diagrams of the Maxwellian view system, including the imaging principle and two distinct
forms derived from it, partial reflector and lightguide structure.

Figure 4. Schematic plots of major microdisplays and combiners. The microdisplays cover liquid-
crystal-on-silicon (LCoS), digital light processer (DLP), laser beam scanner (LBS), micro organic light-
emitting diode (μOLED), and micro light-emitting diode (μLED), while the combiners include freeform
half mirror, birdbath, freeform prism, off-axis holographic optical element (HOE), cascaded mirrors, and
grating couplers. Three kinds of grating couplers are also highlighted: surface relief grating (SRG),
volume Bragg grating (VBG), and polarization volume grating (PVG).

Tables
Table 1. Comparison among AR display light engines.
Display Maturity
Brightness
(nits)
Light
efficiency Form factor
Optical system
complexity
Contrast
Ratio
LCoS
High
10
4
-
10
5
Low
Large
Medium
~10
3
:1
DLP
High
10
4
-
10
5
Medium
Medium
Medium
~10
3
:1
μ
OLED
Medium
10
3
-
10
4
High
Small
Low
~10
4
:1
μ
LED
Low
10
5
-
10
6
High
Small
Low
~10
5
:1
LBS
Medium
>10
4
High
Small
High
~10
5
:1

Table 2. Comparisons among AR optical combiners.
Type Combiner Efficiency*
Form factor
δn Bandwidth FOV diagonal**
Reflective
Freeform mirror
<50%
Large
-
Large
90°
Freeform prism
<50%
Large
-
Large
12
0°
Birdbath
<25%
Large
-
Large
5
2
°
Cascaded mirrors
<20%
Medium
-
Large
4
0°
Diffractive
Off
-
axis HOE
<
2
0%
Small
Small
Small
15°
Traditional VBG
<10%
Medium
Small
Small
40°
HPDLC
<10%
Medium
Medium
Medium
50°
SRG
<10%
Medium
High
Large
5
2
°
PVG
<10%
Medium
Medium
Medium
50°
*These typical values depend on lightguide design.
**These typical values come from products and prototypes.

Highlights
• Introducing the fundamentals of emerging augmented reality (AR) and virtual reality (VR)
technologies and their potential applications
• Analyzing the technical challenges of AR and VR displays
•
Presenting potential solutions to overcome these challenges case by case