Access to this full-text is provided by Optica Publishing Group.
Content available from Optics Express
This content is subject to copyright. Terms and conditions apply.
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42674
Learned phase mask to protect camera under
laser irradiation
JUNY U ZHAN G,1,2,†QING YE,1,2,†YUNLONG WU,1,2
YANGLIANG LI,1,2 YIH UA HU,1,2 AND HAOQ I LUO1,2,*
1State Key Laboratory of Pulsed Power Laser Technology, National University of Defense Technology,
Hefei 230037, China
2Advanced Laser Technology Laboratory of Anhui Province, Hefei, Anhui 230026, China
†The authors contributed equally to this work.
* luohaoqi@mail.ustc.edu.cn
Abstract:
The electro-optical imaging system works under focus conditions for clear imaging.
However, under unexpected laser irradiation, the focused light with extremely high intensity
can easily damage the imaging sensor, resulting in permanent degradation of its perceptual
capabilities. With the escalating prevalence of compact high-performance lasers, safeguarding
cameras from laser damage presents a formidable challenge. Here, we report an end-to-end
method to construct the wavefront coding (WFC) imaging systems with simultaneous superior
laser protection and imaging performance. In the optical coding part, we employ four types
of phase mask parameterization methods: pixel-wise, concentric rings, linear combinations
of Zernike bases, and odd-order polynomial bases, with parameters that are learnable. In the
algorithm decoding part, a method combined of deconvolution module and residual-Unet is
proposed to furthest restore the phase-mask-induced image blurring. The optical and algorithm
parts are jointly optimized within the end-to-end framework to determine the performance
boundary. The governing rule of the laser protection capability versus imaging quality is revealed
by tuning the optimization loss function, and the system database is established for various
working conditions. Numerical simulations and experimental validations both demonstrate that
the proposed laser-protection WFC imaging system can reduce the peak single-pixel laser power
by 99.4% while maintaining high-quality imaging with peak signal-to-noise ratio more than
22 dB. This work pioneers what we believe to be a new path for the design of laser protection
imaging systems, with promising applications in security and autonomous driving.
© 2024 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Electro-optical imaging systems play a pivotal role in the contemporary information society, being
widely employed in both daily life and scientific research endeavors. The imaging system generally
works under focus condition to capture clear image. If the imaging system is unfortunately
irradiated by laser, there will be extremely high intensity light spot on the image plane sensor.
However, imaging sensors such as complementary metal-oxide-semiconductor (CMOS) or
charge-coupled devices (CCD) are sensitive to high-intensity light, which can lead to irreversible
damage when exposed to laser environment [1]. In recent decades, rapid advancements in laser
technology have made low-cost, compact and high-power laser sources widely accessible [2].
Within this context, lasers can be maliciously used to disrupt or permanently damage cameras in
diverse scenarios such as autonomous vehicles and drones [3–6]. To protect imaging system
from laser damage, previous studies have explored techniques including multi-channel spectral
compensation [7], liquid crystal-based switchable optical devices [8], phase-change material
optical limiters [9], metamaterials [10], and integration time adjustment [11]. Despite their
contributions, current technologies based on limiting principles still encounter challenges in
#539988 https://doi.org/10.1364/OE.539988
Journal © 2024 Received 21 Aug 2024; revised 6 Oct 2024; accepted 25 Oct 2024; published 6 Nov 2024
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42675
achieving instantaneous and high dynamic range laser protection while maintaining real-time
imaging.
Wavefront coding (WFC) is a computational imaging technique that combines optical encoding
with post-processing algorithms for decoding. It is originally proposed to extend depth of field
[12], and has recently been demonstrated to reduce the peak intensity of laser on the image plane
sensor, thereby diminishing the risk of laser damage and maintaining real-time imaging [13–18].
However, the WFC laser protection designs still hold potential for improvement. On one hand,
the simplistic phase functions limit the search space for peak performance. On the other hand,
the optical elements and post-processing algorithm are developed separately, which disregards
the balance between laser protection capability and imaging quality, restricting the search for
a globally optimal solution. Therefore, the exploration of complex phase distributions and the
matching of the optical system with the decoding algorithm are essential in achieving optimal
laser protection capability and imaging quality, which can help determine the performance
limits of WFC imaging system in laser protection scenario. Fortunately, deep optics provides a
comprehensive framework for jointly designing optical systems and digital image processing
algorithms in an end-to-end manner [19–24]. By integrating differentiable imaging simulations
with deep learning image processing algorithms, such as Unet [25], the framework has showcased
exceptional performance across a variety of tasks, including extended depth of field [19],
achromatic imaging [20], high dynamic range imaging [21], spectral imaging [22], monocular
depth estimation [23], and privacy protection [24]. The gradient descent-based method for jointly
optimizing optics and algorithms holds the potential to unlock the performance limitations of
WFC in the realm of laser protection.
In this study, we propose an end-to-end approach for the design of laser protection imaging
systems, which involves the balance between laser protection capability and imaging quality
through joint optimization of optical phase and reconstruction algorithms. We explore four
different parameterization schemes for phase design: pixel-wise (PW), concentric rings (CR),
linear combinations of Zernike bases (Zer), and odd-order polynomial bases (OP). Leveraging
prior knowledge of PSF, we develop a hybrid algorithm for high-quality image reconstruction,
which combines Wiener filters with learnable parameters and a concatenated network of residual-
Unet (Res-Unet) [26]. In order to effectively address the diverse application scenarios with
varying laser threat level and imaging requirement, we assign different weights to laser protection
and imaging losses in loss function and investigate the change rule of laser protection capabilities
versus imaging quality for different phase parameterization schemes. Finally, the experimental
validation is performed to demonstrate the efficacy of our proposed designing framework.
2. Theoretical model
2.1. Optical model
Figure 1sketches the laser protection scenario. The imaging system consists of a phase mask,
an imaging lens with focal length f, and a sensor located at distance ffrom the lens to capture
the target scene in real time. There coexists a disruption laser with wavelength of
λ
within the
field of view, and the imaging system faces the risk of damage under laser irradiation. The light
entering the imaging system comprises both incoherent background and coherent laser field.
Placing an appropriate phase mask at the pupil can encode the incoherent scene while modifying
the laser energy distribution on the focal plane sensor, which decreases the maximum single-pixel
power and thereby protects the imaging system form laser damage. The model assumes that the
laser source and imaging system are far apart, much greater than the Rayleigh range
zR=πω2
0/λ
,
where
ω0
represents the beam waist of the laser [13,27]. Consequently, before reaching the pupil
of the imaging system, both the coherent laser and incoherent light from the background scene
can be considered as uniform plane waves. After coherent laser field passing through the phase
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42676
mask and lens, the complex amplitude can be expressed as:
˜
U2(x,y)=A(x,y)˜
U1(x,y)ei(φl+φ). (1)
Fig. 1.
Schematic of the laser protection scenario. The imaging system captures a distant
scene while an unexpected disruptive laser coexists in the field of view. The system is
protected from laser damage by inserting a phase mask at the optical pupil of the imaging
system.
Here,
(x,y)
is the coordinates of the pupil plane. A
(x,y)
is a binary circular mask with a
diameter D, representing the pupil with finite aperture for which light outside the circular region
is completely blocked.
U1(x,y)
is the complex amplitude of the laser on the pupil plane, which is
uniform due to the plane-wave setting.
ϕl=−
i
k
2f(x2+y2)
denotes the phase delay introduced by
the lens, and ϕfor the extra phase delay introduced by the phase mask.
Given the simultaneous need for laser protection and high-quality imaging, it can be challenging
to find an analytical solution of phase mask [15]. Therefore, we devote to obtain the numerical
phase distributions. Various parameterization forms of phase mask have been proposed to better
accommodate specific imaging requirements [28]. Four varieties of the most common phase
masks are utilized in this study, and their optical characteristics are investigated comprehensively,
as indicated in the blue region of Fig. 2. The first is PW, which employs pixel-by-pixel optimization
approach [19]. The PW phase mask is divided into N
×
Ndiscrete pixels, and the phase of each
pixel is an optimized variable, providing the highest degree of freedom. PW has demonstrated
outstanding performance in high dynamic range imaging and super-resolution imaging [19,21].
The second is CR, which divides the phase mask into Nconcentric rings, with the phase of each
ring being optimized as a variable. The phase function of CR phase mask can be represented as:
ϕ=
M
m=1
bm[Circm(r) − Circm−1(r)], (2)
where r
=x2+y2
is the radius, Cir
cm(
r
)=
circ
(r/rm)
,Cir
c0(
r
)=
0.
rm=
md,m
=
1, 2, 3
·· ·
,
dis the ring width, and
bm
denotes the phase for individual ring. CR has shown excellent
performance in monocular depth estimation [29] and diffractive chromatic aberration correction
[20]. The third is Zer, which utilizes a linear combination of Zernike bases [30] to represent the
phase mask, written as:
ϕ=
30
j=1
ajZj, (3)
where
Zj
represents the
jth
Zernike polynomial (for details, see Appendix A), and
aj
is a learnable
variable. Zernike bases form a set of orthogonal continuous function sequences, with each basis
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42677
used to describe a wavefront aberration. Zer corresponds to a smooth phase surface and has found
wide applications in monocular depth estimation, 3D object detection [23], privacy protection
[24], and also in reducing the peak power of lasers on the focal plane [13]. The design of finite
polynomials here is used to achieve a balance between performance and complexities of design
and fabrication. The fourth is OP, which writes phase mask with odd-powered polynomials:
ϕ=
10
i=1
ai(x2i+1+y2i+1), (4)
where
ai
is a learnable variable. OP has shown good performance in extended depth of field
applications [31], and is also used for laser protection in imaging systems [17].
Fig. 2.
The end-to-end framework consists of an imaging model and a CNN-based image
reconstruction network. The imaging model consists of incident light, coding phase, lens and
imaging sensor, with the coding phase denoted by PW, CR, Zer and OP, respectively. The
image reconstruction network consists of a Wiener filter with learnable noise regularization
parameter and a Res-Unet.
The coherent incident light passes through the phase mask and lens, undergoes diffraction, and
reaches the sensor. The complex amplitude of laser spot on the sensor can be expressed as:
˜
U(x′,y′)=F−1{F [ ˜
U2(x,y)]H}, (5)
where
(x′,y′)
is the coordinates of the sensor plane, (
F−1
)
F
denotes the (inverse) Fourier
transform,
H(fx,fy)=ei2π
λLe−iπλL(f2
x+f2
y)
is the Fresnel transfer kernel,
fx
and
fy
denote the
frequency domain coordinates, Lis the distance from the lens to the sensor, and L
=
f. The
normalized laser intensity distribution on the sensor surface can be expressed as
plaser =
|˜
U(x′,y′)|2/|˜
U(x′,y′)|2dx′dy′
. Since the incident laser is considered as a plane wave, the
incoherent point spread function (PSF) of the WFC system pis consist with the laser intensity
distribution plaser.
The yellow area in Fig. 2illustrates the imaging process, where the captured image
Is
is
represented as:
Is=p∗I+η. (6)
Here Iis the intensity of incoherent background scene and ηis the sensor readout noise with
Gaussian noise η∼N(0, σ2).
2.2. Image reconstruction algorithm
After obtaining the encoded image, a decoder needs to be constructed to recover the high-quality
image. In order to achieve superior laser protection performance, it is essential that the PSF
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42678
of imaging system is designed to be highly dispersed. Therefore, incorporating network and
non-blind deconvolution using the PSF as a prior information would be a good choice [32]. The
structure of our image reconstruction network is shown in the gray area in Fig. 2. To reduce
the difficulty of recovering the image by the network, we adopt a Wiener filter with a learnable
noise regularization parameter K[19,32] to deconvolve the coded image first. The parameter
Kis a scalar, whose value is determined through optimization to improve overall performance
across different noise levels. To further mitigate the artifacts introduced by the Wiener filter,
we utilize a residual learning network known as Res-Unet, which has been widely employed in
image deblurring tasks [20,33]. This network enables the learning of differences between the
input image and the real image, thereby facilitating the recovery of high-frequency information.
Specifically, the integral decoding network can be described as:
ˆ
I=fDE[fwiener (Is,p)]. (7)
Here,
ˆ
I
is a clear image of the network prediction.
fwiener(Is,p)
denotes the image deconvolved
using the Wiener filter, which can be written as:
fwiener(Is,p)=F−1F (p)∗F(Is)
|F (p)|2+K. (8)
fDE(·)
represents the Res-Unet network, which takes the output of the Wiener filter as input, and
it produces the predicted clear image. Figure 3depicts the network detail. We configure four
layers in both downsampling and upsampling stages and an intermediate layer between the two
stages. Each layer is a residual convolutional block, and Leaky ReLU is used as the activation for
each layer. We connect the output of the last layer with the input of the network by residuals to
get the final predicted image.
Fig. 3.
The structure of Res-Unet reconstruction network. It consists of nine residual
convolutional blocks, with four blocks using conv for downsampling and another four using
tconv for upsampling. Long connections are added between blocks with the same filter size.
The output channels of the final convolutional block match those of the reconstructed image,
and the output is generated by establishing residual connections with the input.
2.3. End-to-end optimization framework
To determine the appropriate phase function and its post-processing network, we consider
optical coding with algorithmic decoding as a whole system for global optimization, and the
corresponding end-to-end designing framework is shown in Fig. 2. The loss function plays a
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42679
pivotal role in the framework, dictating the performance tendency of the entire system. The
trained loss function here consists of three components: image reconstruction loss
Limg
, laser
protection loss Llaser and energy distribution constraint loss Lenergy, which is written as:
L=αLimg +βLlaser +Lenergy, (9)
where
α
and
β
represent the weights of imaging quality and laser protection capability, respectively.
We use the structural similarity index measure (SSIM, see Appendix B) and
L1
loss for evaluating
the difference between the recovered image and the ground truth, which is expressed as:
Limg =1
M∥I−ˆ
I∥1+[1−SSIM(I,ˆ
I)], (10)
where Mrepresents the number of pixels in the image. The smaller the imaging loss, the higher
the imaging quality.
Llaser
is the laser protection loss for evaluating the sensor damage risk by
the laser:
Llaser =20 max(plaser), (11)
where
plaser
represents the normalized intensity distribution of the laser on the sensor, correspond-
ing to the laser spot profile under unity-power incidence, and the actual intensity is proportional
to the incident laser power. Regardless of the incident laser power, the lower maximum intensity
of laser spot is always preferable according to the design objective of minimizing the risk of laser
damage. Therefore, the smaller the laser protection loss, the lower the maximum intensity of
laser spot, indicating better capability to protect sensor against laser. It should be noted that the
coefficient of 20 in the loss function is artificial and has no specific physical meaning, which can
be regarded as default setting of the imaging system that ensures the balanced imaging and laser
protection capabilities at α:β=1 : 1. In addition, unlike previous end-to-end frameworks, we
require sufficient dispersion of energy on the sensor, resulting in a PSF of extremely large size.
To reduce information loss in edge of the sensor [13], we constrain the energy distribution of the
PSF to ensure it distributed in limited spaces [20]:
Lenergy =− W(x,y)p(x,y)dxdy. (12)
Here, W
(x,y)
is a binary mask, which specifies the spatial distribution range of the PSF. We
define the binary mask W
(
x,y
)=
1(x1<x≤x2)and(y1<y≤y2)
0others
, and we set
x1=y1=−
100
and
x2=y2=
100 to ensure that the spatial distribution of PSF is constrained within a 200
×
200
rectangular region in the center of the sensor.
3. Simulation results and analysis
3.1. Simulation setup
Simulation adopts a monochrome imaging system with focal length of 50
mm
and aperture
diameter of 4.86
mm
. The size of sensor pixel is 5.86
µm
, with a resolution of 830
×
830. Phase
mask is discretized on a grid size of 830
×
830, with feature size of 5.86
µm
. The imaging
system works at the wavelength of 633 nm. Sensor readout noise is Gaussian with standard
deviation drawn from a uniform distribution between 0.001 and 0.01 (with an image scale of
[0,1]). Training data is sourced from the DIV2 K single-image super-resolution dataset [34].
Data are randomly cropped to a size of 256
×
256 and converted to grayscale. 759 images are used
for training and 50 for testing. We implemented the proposed framework in PyTorch, utilizing its
automatic differentiation for error backpropagation, enabling end-to-end training of optical phase
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42680
and reconstruction network parameters. Specifically, we trained the entire model for 50 epochs
using the Adam optimizer with a batch size of 1, employing step learning rate decay every 10
epochs with a decay factor of 0.9. Model training was conducted on a single NVIDIA GeForce
RTX 3080ti GPU, taking approximately 12 hours to complete a full model optimization.
3.2. Evaluation of four types of phase masks at weight 1:1
First, we design the WFC imaging systems with four phase parameterizations via end-to-end
framework with equal weighting given to protection and imaging (
α
:
β=
1 : 1) and analyze
the laser suppression capability and imaging quality. All phase masks are initialized to zero
before optimization, corresponding to conventional imaging system. To quantitatively assess the
improvement in laser protection capability compared to conventional imaging system, we define
the laser suppression ratio (LSR):
LSR =max(p0)
max(plaser), (13)
where
plaser
and
p0
represent the laser intensity distributions on the sensor surface with and
without phase mask modulation, respectively. LSR is the ratio of the highest single-pixel power
of the laser in sensor plane of a conventional imaging system to that after phase modulation.
Higher LSR indicates better laser protection capability of the WFC system and thus lower risk
of damage under laser irradiation. Moreover, the final optimized model was evaluated on 50
test images, and the peak signal-to-noise ratio (PSNR) and SSIM of the recovered images was
calculated to assess the system imaging quality, as shown in Table 1. The definitions of PSNR
and SSIM are given in Appendix B. For more visual comparison, Fig. 4illustrates the phase
distribution and PSF corresponding to each phase model, as well as two examples selected from
the test set. It can be observed in Fig. 4(b) that the laser facula modulated by four phase masks
are all significantly larger than the Airy disk of conventional imaging system, thereby spreading
energy, reducing peak intensity and realizing protection against laser. Meanwhile, the coded
images captured by WFC imaging systems [Figs. 4(c) and 4(e)] can be restored effectively using
our reconstruction algorithm. The decoded images exhibit high quality comparable to those of
conventional system, as shown in Figs. 4(d) and 4(f). Notably, although designed with the same
weight ratio of imaging and protection, both PSNR and LSR are distinct for the four phase masks.
Among them, PW possesses both the utmost imaging quality (PSNR
=
24.19 dB) and laser
protection capability (LSR
=
186.6), attributed to its highest optimization degrees of freedom,
as shown in Fig. 4(a). Design freedom of the other three phase masks is comparatively lower,
leading to a reduction in peak performance and exhibiting the decreased PSNR and LSR values.
Table 1. Quantitative evaluation of four optics models at wteight 1:1.
Phase
masks
PSNR/SSIM LSR
Measurement Ours Ours w/o
Res-Unet
Ours w/o Wiener
deconvolution
DeblurGAN
[35]
PW 16.87/0.39 24.19/0.85 22.92/0.53 20.54/0.80 20.30/0.74 186.6
CR 15.60/0.39 22.99/0.81 22.56/0.53 20.78/0.79 18.48/0.70 147.1
Zer 17.94/0.42 22.95/0.82 22.31/0.51 20.55/0.78 19.84/0.70 100.6
OP 14.59/0.35 22.45/0.79 21.78/0.51 19.35/0.73 18.14/0.64 126.1
To further demonstrate the advantages of our optical design and reconstruction network,
we conducted an ablation study comparing our network with our network without Res-Unet,
our network without Wiener deconvolution, and DeblurGAN [35]. It should be noted that
DeblurGAN, as a representative of classical blind deblurring networks, is used independently.
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42681
Fig. 4.
Examples of different phase masks evaluated in the simulation. (a) Phase distributions
of the four phase masks. (b) corresponding PSFs. (c)-(f) original measured images and
network reconstructed images. The LSR of the corresponding phase and the PSNR of the
reconstructed image are shown in the bottom of the figure.
The results are detailed in Appendix Cand quantitatively summarized in Table 1. Our findings
indicate that for restoring images encoded with our designed PSF, the non-blind deblurring
method yielded superior results. Notably, combining Wiener deconvolution with Res-Unet
achieved the best deblurring performance. Moreover, we also investigate the performance of our
network for various noise levels and compare it with our network with fixed-Kin deconvolution.
The results are given in Appendix D. Our network show robustness to the noise levels and the
overall performance is higher than the fixed-Knetwork.
To assess the effectiveness of our designed PSF, we compared the PSF derived from the PW
model with a half-ring PSF (HR-PSF) [18]. Using our proposed network for reconstruction,
where only the network was optimized and the PSF remained fixed, we provided quantitative
evaluations at Gaussian noise levels of
σ=
0.003 and
σ=
0.006 in Appendix E. The PW-PSF
exhibits superior protection capability and image quality compared to HR-PSF, indicating that
our designed PSF outperformed the HR-PSF.
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42682
Fig. 5.
Visualization of the checkpoints of four kinds of phase masks during 50 epochs of
training. Considering the lower limit of imaging quality, we show the fraction of PSNR
higher than 20 dB. The trend of each model was fitted using an exponential function. The
gray dashed box was used to compare the LSR of different models for the same imaging
quality, while the data in the gray solid box was zoomed in for better observation.
3.3. Constraints between imaging and laser protection
Pervious section has demonstrated that WFC imaging system, empowered by end-to-end design
framework, can achieve simultaneous high-quality imaging and laser protection. The PSNR
and LSR are fixed for individual phase mask optimized using the same loss function. However,
under laser irradiation of different threat levels, the WFC imaging systems should be adjusted
to achieve a balance between the requirements for laser protection and imaging quality, which
necessitates acknowledge of constraints between imaging and laser protection capabilities, as
well as the design dataset. By tuning the weight ratios α:βin loss function, the WFC imaging
system can feature various performance tendencies through end-to-end optimization.
To reveal the constraint relationship between imaging quality and protection capability, we
set three weight ratios
α
:
β=
5 : 1, 1 : 1, 1 : 5, and train 12 models end-to-end at each
weight ratios. Furthermore, we store the checkpoint corresponding to each epoch during the
training process to establish a dataset for suiting diverse working conditions, which records
the phase distribution and the parameters of image reconstruction network. The parameters
of each checkpoint are employed to the test set and record the corresponding PSNR and LSR.
The dataset not only enables us to observe the entire process of system optimization, but also
to investigate the capability range of different phase masks. Figure 5gathers the checkpoints
throughout the optimization process and present them in the PSNR-LSR synthetic space, where
the points located in close proximity to the top right corner are indicative of superior overall
performance. In order to show the relationship between LSR and PSNR more intuitively, Fig. 5
gives the PSNR-LSR trend lines of the four types of phase masks for visualization (colored solid
lines). Laser protection necessitates a larger PSF, which in turn complicates image recovery
and diminishes imaging quality. Hence, there is a constraint between PSNR and LSR, wherein
LSR reduces as the PSNR increases, as illustrated in Fig. 5. This result indicates that there is no
best solution of WFC imaging system that offers both maximum laser protection and imaging
capabilities. Instead, only locally optimal solution exists for specific requirements. For example,
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42683
Fig. 6.
Examples used to evaluate the performance of different phase masks within
the dashed box in Fig 5. (a) The phase distributions of the four phase masks. (b) The
corresponding PSF. (c)- (f) The original measured image and the reconstructed image. The
LSR of the corresponding phase mask and the PSNR of the reconstructed image are shown
at the bottom of the figure.
when application scenario has minimum demand on the imaging quality, such as PSNR
>
22 dB,
appropriate points can be selected from the PSNR-LSR synthetic space for system design based
on qualification. As shown in the gray dashed box in Fig. 5, where the points all have the almost
same imaging quality, we can pick the point with the highest LSR for each phase mask to achieve
the optimal laser protection (marked by red box).
Figure 6(a) shows the phase distributions for each point picked in Fig. 5, Fig. 6(b) shows the
corresponding PSFs, Figs. 6(c) and 6(e) show the encoded of two examples selected from the test
set and Figs. 6(d) and 6(f) for decoded images. It can be observed that the PSF of PW exhibits
the most dispersed energy, thus realizing a much higher LSR
=
584 than the other three phase
masks (LSR
<
200). Imaging quality of all the four phase masks are almost the same, consistent
with the design requirements.
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42684
4. Experimental validation and analysis
4.1. Experiment setup
For experimental validation of the end-to-end framework, we build a WFC experimental prototype.
Figure 7illustrates the schematic of the experimental setup, which consists of the light source
and imaging module. With this optical path, we can measure both PSF and coded scene for
the WFC imaging system with designed phase masks. The light source comprises of two parts:
one is the point source for PSF measurement, and the other is the target for imaging. Light
emitted from He-Ne laser (633 nm) passes through a spatial filter with 10
µ
m pinhole to form
the point source. The target is illuminated by a red LED, and then reflected into the imaging
modules by a beam splitter. Lens 1 of focal length 1000 mm (Lbtek MAD529-A) collects light
and produces plane wave to simulate the scenario of infinitely far imaging. The imaging module
uses a 1920
×
1080 spatial light modulator (SLM, UPOLabs, HDSLM45R,) to realize phase
modulation, and the pixel size is 4.5
×
4.5
µ
m. Before into the imaging module, a polarizer
convert the light into linearly polarized. Within the imaging module, lenses 1 and 2 of focal
length 75 mm (Lbtek, MAD412-A) form the 4f telescopic system to project the wavefront at
SLM onto the imaging lens 4. A spectral filter (Daheng, GCC-202208) is placed enclosed lens 4
to remove undesirable diffracted artifacts caused by the SLM. The camera consists of an imaging
lens 4 of focal length 50 mm (Lbtek, M5028-MPW3) and sensor of pixel size 5.86
×
5.86
µ
m
(Daheng, MER2-231-41U3 M) to capture the PSF and coded image.
Fig. 7.
Schematic of Laser protection experimental prototype which is capable of recording
both the PSF and the encoded image.
4.2. Results and discussion
Figure 8(a) presents the loaded phase distribution in WFC experimental system, corresponding to
the four end-to-end optimized phase masks marked in the gray dashed box in Fig. 5. Figure 8(b)
depicts the measured PSFs, which exhibits good consistency in size and morphology with
simulated results in Fig. 6(b), including the conventional and WFC imaging systems, validating
the accuracy of the simulation. Figures 8(c) and 8(e) present the images directly captured by the
camera. The conventional system produces clear image while that of WFC system is severely
blurred due to the expanded PSF. Figure 8(d) depicts the restored images by using the optimized
decoding algorithm for the first scene, wherein the details such as digits, letters and car door
lines are all effectively restored. Figure 8(f) shows restored images for the second scene with
dense grilles, and the high-frequency information is also well recovered. The superior image
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42685
quality results from the effective combination of Wiener filtering for deconvolution and Res-Unet
for refinement. The average PSNR of the two-scene restored images exceeds 22.4 dB for the four
kinds of phase masks, which is consistent with the theoretical design.
Fig. 8.
Experimental verification of imaging quality and laser protection balance for
different phase masks. (a) Phase loaded on SLM. (b) Experimentally measured PSF. (c)-(f)
Measurement and recovery of two different scenes. The experimentally measured LSR is
listed at bottom of PSF figure, and PSNR below recovered images. Due to the inverted
imaging, all captured images are vertically flipped for clarity.
Table 2shows the experimentally measured LSR for WFC system with the four optimized
phase masks. It can be observed that the measured LSR exhibits the identical trend as analyzed
in the simulation. PW possesses highest laser protection capability, followed by CR and OP, with
Zer being the lowest. The pixel size of phase mask in simulation is set same as that of the sensor
(5.86
µ
m) to ensure the accuracy of the calculated energy distribution, which is sampled to 4.5
µ
m
to be loaded on the SLM in experiment. Because the designed phase mask feature fine details,
especially the PW, the sampling slightly changes the PSF, enlarging power of several pixels and
thus reducing the measured LSR. Therefore, there exists a different degree of degradation for
the measured LSR compared to simulation, and the detail analysis can be found in Appendix
F. Nevertheless, three of our design phase masks (PW, CR and OP) render laser suppression
capability of approximately two orders of magnitude in experiments, with PW able to reduce the
peak laser energy received by a conventional imaging system by up to 99.4%, which will greatly
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42686
reduce the risk of damage to the imaging sensor under laser irradiation. Finally, we also give the
imaging result that combines the laser spot and the scene by both simulation and experiments
(see Appendix G). The results show that the majority of image information can be effectively
restored by our network, expect for the laser interference region on the sensor.
Table 2. Comparison of simulated and experimentally measured LSR.
Optics models PW CR Zer OP
simulation 584 167.3 109 127.9
experiment 174 96 66.2 93.6
5. Summary and outlook
In summary, we propose an end-to-end framework for simultaneously optimizing phase mask
and reconstruction algorithms, ensuring high-quality imaging while minimizing the risk of laser
damage to imaging sensors. The end-to-end framework evaluates PW, CR, Zer and OP phase
masks, offering the performance range and design dataset. Result indicates a trade-off between
laser protection capability and imaging quality in WFC systems, which suggests tailored phase
parameter selection should be based on specific applications. PW possesses maximum design
flexibility and superior performance, validated through numerical simulations and experiments
achieving high-quality imaging (PSNR
>
22 dB) and significant laser suppression (
>
99.4%).
This work introduces a real-time laser protection solution, which may be applied to high-risk
scenarios against laser, such as security and autonomous driving, to enhance imaging system
adaptability. In the next step, this method has potential to be further extended to cover a broader
frequency range by considering the dispersion effect of the phase element. Moreover, the
performance of the imaging system may be further improved through other methods, including
separable model [36], phase retrieval [37] and total variation regularization [38].
Appendix A: mathematical description of Zernike polynomials
We adopt Noll’s notation and numbering scheme [30] which defines Zjin the polar coordinates
as:
Zeven,j=√n+1Rm
n(r)√2cos(mθ)
Zodd,j=√n+1Rm
n(r)√2sin(mθ)
m≠0
Zj=√n+1R0
n(r)m=0
, (14)
Rm
n(r)=
(n−m)/2
s=0
(−1)s(n−s)!
s![(n+m)/2−s]![(n−m)/2−s]!rn−2s(15)
Here nis the power of the radial coordinate rand mis the multiplication factor of the angular
coordinate θ. Both mand nsatisfy m≤nand (n−m)is even.
Appendix B: definition of PSNR and SSIM
The PSNR between images I1and I2can be defined as:
PSNR(I1,I2)=10log10 MAX2
I
MSE(I1,I2). (16)
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42687
Here MA
XI
is the maximum possible pixel value of the image and in this paper MA
XI=
1.
MSE
(I1,I2)=1
mn
m−1
i=0
n−1
j=0[I1(i,j) − I2(i,j)]2
is the mean square error of I
1
and I
2
with image size
m×n.
The SSIM between images I1and I2can be defined as:
SSIM(I1,I2)=(2µ1µ2+c1)(2σ12 +c2)
(µ2
1+µ2
2+c1)(σ2
1+σ2
2+c2), (17)
Here
µ1
is the mean of I
1
and
µ2
for I
2
,
σ1
is the standard deviations of I
1
and
σ2
for I
2
,
σ12
is
the covariance of I1and I2,c1=0.0001 and c2=0.0009(image scale of [0,1]) [39].
Appendix C: comparison of image reconstruction approaches
In Table 1, we quantitatively evaluated the image quality of our recovery network and the image
quality after removing parts in it, and we also compared the proposed approach to DeblurGAN
method [35]. In order to make a more intuitive comparison, we show decoded images under
different PSFs and reconstruction methods in Fig. 9.
Fig. 9.
Comparison of recovered images under different PSFs and reconstructed methods.
(a) PSFs corresponding to different phase models. (b) Ground truth. (c) The measurements
on the sensor. (d)-(g) are reconstructed images respectively from our method, our method
removes Res-Unet, our method removes Wiener deconvolution and DeblurGAN [35].
Appendix D: discussion on Kin Wiener filter
To demonstrate the superiority of our network with learnable K, we quantitatively evaluate the
image quality through the learnable-Knetwork for multiple noise level and compare it with the
network with various fixed K. The PW-PSF used here is consist with that designed in section 3.2.
The parameter Kranges from 0.000002 to 0.002 approximately according to the definition of
ratio between the power spectrum density of the noise and of the scenes [40]. The Kis optimized
as 0.0012 end-to-end, which is inside the above Krange. Since the image quality degrades
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42688
seriously when Kis lower than 0.0002, we choose 0.0002, 0.0008, 0.0014 and 0.002 as the fixed
Kto be used in conjunction with network training, and the results are listed in Table 3. It can be
observed that the optimal Kin the hybrid reconstruction network varies under different noise
levels, and the lower noise level prefers lower K. We also calculate the average of the PSNR and
SSIM for different noise levels to evaluate the overall performance. Although the fixed Kmay
perform better performance than the learned Kat certain noise level, the overall performance of
the learned Kis superior, providing greater robustness against the noise levels.
Table 3. Quantitative evaluation of fixed K and learnable K was performed at different Gaussian
noise levels.
PSNR/SSIM
Noise level σ=0.003 σ=0.006 σ=0.009 mean
Fixed K
0.002 25.74/0.89 24.14/0.85 22.67/0.80 24.18/0.847
0.0014 25.92/0.90 24.18/0.85 22.55/0.79 24.22/0.847
0.0008 26.18/0.90 23.64/0.85 21.77/0.79 23.86/0.847
0.0002 25.30/0.88 21.44/0.82 19.47/0.75 22.07/0.817
Learned K(0.0012) 26.14/0.90 24.13/0.85 22.46/0.80 24.24/0.850
Appendix E: quantitative comparison of our PW-PSF and HR-PSF [18]
To validate the advantages of our designed PSF, we compared the PW-PSF with the HR-PSF
using our reconstruction network. The HR-PSF is known for achieving high-fidelity imaging and
a high LSR. Moreover, the HR-PSF outperforms multi-HR-PSFs (such as the five-half-ring PSF),
which is why we use the HR-PSF as a benchmark for our design. The HR-PSF is implemented
according to Eq. (5) in [18]. Specifically, the PSF remains fixed while the network was optimized
at noise levels of
σ=
0.003 and
σ=
0.006. We tested 50 images in total, and the average PSNR,
SSIM, and LSR are listed in Table 4. Figure 10 presents the encoding and decoding results for
both PSFs within the same scene.
Appendix F: error analysis of experimentally measured LSR
We sample the phase distribution from 5.86
µ
m to 4.5
µ
m and calculate the laser intensity
distribution on the sensor surface by Fresnel diffraction. Since the pixel size of the camera is
larger than the sampling interval of the light field in such configuration, we integrate the laser
intensity in each sensor pixel to find the maximum intensity. However, the position relationship
between the light field and the pixel of the camera is hardly determined in experiment, and several
typical positional relationships are illustrated in Fig. 11, wherein the green box denotes a camera
pixel, the black gridding for the sampling interval of the light field and the filled box indicates the
position of peak intensity. Therefore, there should exists a theoretical interval for the LSR. We
calculate the LSR intervals corresponding to the four phases respectively, as shown in Table 5. It
can be observed that the LSR measured by the experiment is all in the theoretical interval.
Table 4. Quantitative evaluation of PW-PSF and HR-PSF.
Noise Level PSF PSNR SSIM LSR
σ=0.003 PW (ours) 24.74 0.888 630
HR 24.24 0.854 542
σ=0.006 PW (ours) 23.19 0.868 630
HR 23.39 0.846 542
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42689
=
0.006
Fig. 10.
Comparison of reconstruction results of encoded images corresponding to PW-PSF
and HR-PSF under different noise levels.
Fig. 11. Some possible positional relationships between camera pixels and light fields
Table 5. Comparison of LSR at different light field sampling intervals.
ideal analysis experiment
PW 584 [122.6,332.9] 174
CR 167.3 [87.1,165.3] 96
OP 127.9 [75.6,142.3] 93.6
Zer 109 [44.9,102.5] 66.2
Appendix G: images combining the scene and the laser spot
The image combining both laser spot and scene can be written as [41]:
Is=p∗I+αlplaser +η, (18)
where pis the incoherent PSF, Iis the incoherent scene,
αl
is the laser intensity (with unit of
saturation intensity),
plaser
is the laser distribution after energy normalization and
η
is the sensor
readout noise with Gaussian noise
η∼
N
(
0,
σ2)
. The Gaussian noise level
σ=
0.003 and
laser intensity of
αl=
100 are set in simulation. The experimental setup is consistent with that
described in Section 4. The simulated and experimental imaging results that combines both
scene and laser spot for the four types of phase masks are shown in Fig. 12, in which the laser
spot is marked by the red box. The simulated and experimental imaging results coincides with
each other. The majority of image information can be effectively restored by our network, expect
for the laser interference region. It should also be pointed out that this work focuses on the laser
safety of imaging system instead of the glare reduction, and thus the laser spot remains present in
the restored images.
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42690
Fig. 12.
Simulation and experiment in the presence of laser interference. The first two lines
show the simulation results, and the last two lines show the experimental results. (a) PW. (b)
CR. (c) Zer. (d) OP.
Funding.
Postdoctoral Fellowship Program of China Postdoctoral Science Foundation (GZC20233531); Advanced
Laser Technology Laboratory Foundation of Anhui Province (AHL2021QN03, AHL2022ZR03); Research Project
of National University of Defense Technology (ZK2041); Technology Domain Fund of Basic Strengthening Plan
(2021-JCJQ-JJ-0284, 2022-JCJQ-JJ-0237).
Disclosures. The authors declare no conflicts of interest.
Data availability.
Data underlying the results presented in this paper are not publicly available at this time but may
be obtained from the authors upon reasonable request.
References
1.
J. Hu, Z. Xin, L. Shugang, et al., “Image processing method for laser damage probability measurement by single-shot
of laser pulse,” Opt. Express 19(11), 10625–10631 (2011).
2.
D. J. Richardson, J. Nilsson, and W. A. Clarkson, “High power fiber lasers: current status and future perspectives,” J.
Opt. Soc. Am. B 27(11), B63–B92 (2010).
3.
C. Yan, “Can You Trust Autonomous Vehicles: Contactless Attacks against Sensors of Self-driving Vehicle,” Def
Con 24(8), 109 (2016).
4.
Z. Stein and A. Stern, “Imperceptible CMOS camera dazzle for adversarial attacks on deep neural networks,” arXiv
(2023).
5. J. Petit, B. Stottelaar, and M. Feiri, “Remote Attacks on Automated Vehicles Sensors: Experiments on Camera and
LiDAR,” in Proceedings of the Black Hat Europe (2015), pp. 1–13.
6.
Z. Fu, Y. Zhi, S. Ji, et al., “Remote Attacks on Drones Vision Sensors: An Empirical Study,” IEEE Trans. Dependable
and Secure Comput. 19(5), 3125–3135 (2022).
7.
G. Ritt and B. Eberle, “Use of complementary wavelength bands for laser dazzle protection,” Opt. Eng.
59
(1), 1
(2020).
8.
R. Zhang, Z. Zhang, J. Han, et al., “Advanced liquid crystal-based switchable optical devices for light protection
applications: principles and strategies,” Light: Sci. Appl. 12(1), 11 (2023).
9.
H. Guan, F. Ren, S. Liang, et al., “Ultra-High Transmission Broadband Tunable VO2 Optical Limiter,” Laser
Photonics Rev. 17(4), 2200653 (2023).
10.
A. Howes, Z. Zhu, D. Curie, et al., “Optical Limiting Based on Huygens Metasurfaces,” Nano Lett.
20
(6), 4638–4644
(2020).
11.
G. D. Lewis, R. Struyve, C. Boeckx, et al., “The effectiveness of amplitude modulation on the laser dazzling of a
mid-infrared imager,” in Proceedings of the High-Power Lasers and Technologies for Optical Countermeasures
(2022), pp. 120–137.
12.
E. R. Dowski and W. T. Cathey, “Extended depth of field through wave-front coding,” Appl. Opt.
34
(11), 1859–1866
(1995).
13.
J. Wirth, A. Watnik, and G. Swartzlander, “Optimized pupil-plane phase masks for high-contrast imaging,” Appl.
Opt. 57(20), 5688–5693 (2018).
Research Article Vol. 32, No. 24 / 18 Nov 2024 / Optics Express 42691
14. X. Peng, E. F. Fleet, A. T. Watnik, et al., “Learning to See Through Dazzle,” arXiv (2024).
15.
J. H. Wirth, “Point Spread Function and Modulation Transfer Function Engineering,” Ph.D. dissertation (Rochester
Institute of Technology, 2019).
16.
G. Ruane, A. Watnik, and G. Swartzlander, “Reducing the risk of laser damage in a focal plane array using linear
pupil-plane phase elements,” Appl. Opt. 54(2), 210–218 (2015).
17.
Q. Ye, Y. Wu, H. Zhang, et al., “Experimental damage thresholds of a laser suppression imaging system using a
cubic phase plate,” Chin. Opt. Lett. 21(4), 041403 (2023).
18.
J. H. Wirth, A. T. Watnik, and G. A. Swartzlander, “Half-ring point spread functions,” Opt. Lett.
45
(8), 2179–2182
(2020).
19.
V. Sitzmann, S. Diamond, Y. Peng, et al., “End-to-end optimization of optics and image processing for achromatic
extended depth of field and super-resolution imaging,” ACM Trans. Graph. 37(4), 1–13 (2018).
20.
X. Dun, H. Ikoma, G. Wetzstein, et al., “Learned rotationally symmetric diffractive achromat for full-spectrum
computational imaging,” Optica 7(8), 913–922 (2020).
21.
C. A. Metzler, H. Ikoma, Y. Peng, et al., “Deep Optics for Single-Shot High-Dynamic-Range Imaging,” in Proceedings
of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), pp. 1375–1385.
22.
R. A. Carrascal, P. Gomez, and H. Arguello, “Middle output regularized end-to-end optimization for computational
imaging,” Optica 10(11), 1421–1431 (2023).
23.
J. Chang and G. Wetzstein, “Deep Optics for Monocular Depth Estimation and 3D Object Detection,” in Proceedings
of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019), pp. 10193–10202.
24.
C. Hinojosa, J. C. Niebles, and H. Arguello, “Learning Privacy-preserving Optics for Human Pose Estimation,” in
Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2021), pp. 2573–2582.
25.
O. Ronneberger, P. Fischer, and T. Brox, “U-net: convolutional networks for biomedical image segmentation,” in
Medical Image Computing and Computer-Assisted Intervention (MICCAI) (2015), pp. 234–241.
26.
Z. Zhang, Q. Liu, and Y. Wang, “Road Extraction by Deep Residual U-Net,” IEEE Geosci. Remote Sensing Lett.
15(5), 749–753 (2018).
27. O. Svelto, Principles of Lasers (Springer, 2010), Chap. 4.
28.
X. Liu, L. Li, X. M. Liu, et al., “Investigating deep optics model representation in affecting resolved all-in-focus
image quality and depth estimation fidelity,” Opt. Express 30(20), 36973–36984 (2022).
29.
H. Ikoma, C. M. Nguyen, C. A. Metzler, et al., “Depth from Defocus with Learned Optics for Imaging and
Occlusion-aware Depth Estimation,” in 2021 IEEE International Conference on Computational Photography (ICCP)
(2021), pp. 1–12.
30. R. J. Noll, “Zernike polynomials and atmospheric turbulence,” J. Opt. Soc. Am. 66(3), 207–211 (1976).
31.
Y. Li, Y. Lyu, J. Wang, et al., “Optimizing wavefront coding for extended depth of field: a synchronous algorithm for
optical element and decoding optimization,” Opt. Lett. 48(22), 5847–5850 (2023).
32.
K. Yanny, K. Monakhova, R. W. Shuai, et al., “Deep learning for fast spatially-varying deconvolution,” Optica
9
(1),
96–99 (2022).
33.
S. Tan, Y. Wu, S.-I. Yu, et al., “CodedStereo: Learned Phase Masks for Large Depth-of-field Stereo,” in Proceedings
of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021), pp. 7170–7179.
34.
E. Agustsson and R. Timofte, “NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2017), pp.
126–135.
35.
O. Kupyn, V. Budzan, M. Mykhailych, et al., “DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial
Networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
(2018), pp. 8183–8192.
36.
S. S. Khan, V. Sundar, V. Boominathan, et al., “FlatNet: Towards Photorealistic Scene Reconstruction From Lensless
Measurements,” IEEE Trans. Pattern Anal. Mach. Intell. 44(4), 1 (2020).
37.
B. Zhang, X. Yuan, C. Deng, et al., “End-to-end snapshot compressed super-resolution imaging with deep optics,”
Optica 9(4), 451–454 (2022).
38.
J. Wu, H. Zhang, W. Zhang, et al., “Single-shot lensless imaging with Fresnel zone aperture and incoherent
illumination,” Light: Sci. Appl. 9(1), 53 (2020).
39.
Z. Wang, A. C. Bovik, H. R. Sheikh, et al., “Image quality assessment: from error visibility to structural similarity,”
IEEE Trans. on Image Process. 13(4), 600–612 (2004).
40.
R. C. Gonzalez, R. E. Woods, and S. L. Eddins, Digital Image Processing Using MATLAB (Prentice Hall, 2004),
Chap. 4.
41.
Y. Wu, Q. He, T. Xue, et al., “How to Train Neural Networks for Flare Removal,” in Proceedings of the IEEE/CVF
International Conference on Computer Vision (ICCV) (2021), pp. 2239–2247.
