Recent Developments in coded aperture multiplexed imaging systems
A. Mahalanobis, C. Reyner, T. Haberfelde
Lockheed Martin Missiles and Fire Control, 5600 Sand Lake Road, Orlando, Florida, 32819
Mark Neifeld, University of Arizona
B.V.K. Vijaya Kumar, Carnegie Mellon University
We will review recent developments in coded aperture techniques for unconventional imaging
systems. Specifically, we are interested in looking simultaneously in multiple directions using a
common aperture. To accomplish this, we interleave several sparse sub-apertures that are pointed in
different directions. The goal is to optimize the sub-apertures so that the point spread function (PSF)
is well behaved, and resolution is preserved in the images. We will present an analysis of the
underlying PSF design concept, as well as the necessary phase optimization techniques.
The problem of wide area persistent surveillance can be addressed by designing a sensor that looks
down from a very high altitude, and continuously images over a large area. The challenge is to
achieve the resolution needed to detect and track objects of interest. Under the LACOSTE program,
we have developed a concept that employs interleaved sub-apertures with multiplex look directions
to achieve resolution while simultaneously maintaining wide area coverage. Another key constraint
that has guided our system design is the desire to use a minimum number of focal plane arrays
(FPAs) to reduce the cost as well as size, weight, and power requirements of the sensor.
The idea of using computational optics  and point spread function engineering  has been
reported by many researchers. These techniques offer powerful methods for designing and
optimizing non-conventional imaging systems. We use an image multiplexing approach that is
multi-scale hierarchical, and relies on image coding in the Fourier plan. The basic approach is
shown in Figure 1, although the exact dimensions and numbers of components vary with the design.
The overall Mask consists of several apertures, each of which is dedicated to address a section of
the ground. In turn, each aperture consists of several interleaved sub-apertures that address some
part of the assigned section at full resolution. Thus in principle, a small region of the ground can be
sensed with the FPA simply by looking through one such sub-aperture. However to gather sufficient
photons, it becomes necessary to simultaneously look through several sub-apertures and image
several sections of the ground on the FPA. The trick is to then demultiplex the different images 
and put them back together to create the large scene.
Visual Information Processing XVII, edited by Zia-ur Rahman, Stephen E. Reichenbach, Mark Allen Neifeld,
Proc. of SPIE Vol. 6978, 69780G, (2008) · 0277-786X/08/$18 · doi: 10.1117/12.781462
Proc. of SPIE Vol. 6978 69780G-1
2008 SPIE Digital Library -- Subscriber Archive Copy
Figure 1: In the basic approach, the ground is divided into several sections that are addressed by an array of
independent apertures. In turn each aperture has a set of interleaved sub-apertures that multiplex several
look directions on the same FPA. Image decoding techniques are used to separate the look directions and
reconstruct the original image
To achieve the image coding and simultaneously look in multiple directions, we need an active
device such as the “Eyelid” array  shown in Figure 2. These eyelids act as tiny shutters that can
be operated in groups to open or close different sub-apertures. The eyelid actuator consists of
alternating polymer and metal layers forming a curled film that is anchored to a substrate. The
device also contains a fixed conductor deposited on the substrate. Electrostatic actuation is achieved
by applying an electric field between the metal electrode in the flexible film stack and the fixed
conductor that unrolls the flexible film over the substrate surface. The curl in the device is caused
by thermal expansion differences between the metal and polymer layers in the thin film stack and
can be varied by changing the thin-film layer thicknesses.
The aperture is tiled with an array of eyelids that are divided into groups, each dedicated to look at a
different region on the ground. This is achieved by using interleaved beam-steering functions that
causes each group of eyelid to “look” at a unique direction as shown in Figure 3. The beam-steering
function can be realized using diffractives, various types of prisms, and other optical strategies that
offer different advantages in system trade-studies. The resulting images can be enhanced to remove
the effects of sparse aperture using well known techniques for image restoration .
The rest of the paper is organized as follows. Section 2 discusses the concept of interleaving bulk
prisms across an aperture to simultaneously look in multiple directions. Section 3 presents the
results of simulations that show it is possible to use small phase delays (shims) to optimize the PSF.
A summary and direction for future work are presented in Section 4.
Proc. of SPIE Vol. 6978 69780G-2
2. Beam Steering using Interleaved Prisms
At the heart of the system described in Section 1 is a set
of interleaved sparse apertures, each of which is
dedicated to look in a particular direction. Although
there are several methods for steering look directions, a
conceptually easy approach to point the optical axis in a
particular direction is to use bulk prisms. Consider the
case shown in Figure 3(a) where the aperture looks in
one of two directions (one at a time) using two different
bulk prisms. However, for the aperture to simultaneously
look in both directions, the prisms have to be interleaved
into a common surface as shown in Figure 3(b).
Essentially, complimentary sections of the two prisms
may be fitted together creating a surface that follows the
profile of one or the other prism. In Figure 3(b), we also
see that individual prism sections can be thought of as a small triangular micro-prism at the top, and
a rectangular base or shim whose height depends on the section of the bulk prism from which the
piece is obtained.
Figure 3: An aperture can be made to look separately in different directions using bulk prisms (a). The
interleaved arrangement in (b) allows the aperture to look in both directions simultaneously. The sections of
the bulk prisms in (b) can be thought of as a small mircro-prism on top of a shim.
The benefit of using a bulk prism is that over a narrow band of frequencies, the look direction is
nearly independent of the wavelength and no chromaticity effects are observed.. The problem of
interleaving bulk prisms is also conceptually simple, but unfortunately difficult to realize in practice
partly due to large differences between the heights of the shims of adjacent pieces. Therefore it is of
interest to reduce these while maintaining the quality of the PSF. This idea is shown in Figure 4
Figure 2: An example of a small array of
eyelids produced by RTI that are used to
dynamically control the sub-aperture and
its look directions.
Proc. of SPIE Vol. 6978 69780G-3
where segments of the bulk prism that all point in the same direction are replaced by identical
micro-prisms with considerably smaller shims than the original sizes based on the bulk prism’s
profile. The goal is to optimize the choice of the smaller shims so that resulting phase approximates
that of the original bulk prism. The trade however is that in making the shorter shims, chromaticity
effects become pronounced and must be factored into the analysis.
Figure 4: The sections of the bulk prism on the left are replaced by micro-prisms and smaller shims.
2.1. Optimizing phase
The question arises how best to optimize the choice of the smaller shims so that the effective phase
across the aperture behaves like that of the bulk prism. Let us assume that N sections of a bulk
prism are distributed randomly across the aperture. Each section of the bulk prism is modeled as a
micro-prism with slope η and a base of dimension a. We apply a weight
α which represents the
α is 1.0, although complex phase determined by the size of the attached shim. The magnitude of
particular sections of the aperture can be eliminated from the optimization by choosing the
α to be 0. Using n as the refractive index of the material, and λ as the wavelength,
the transmittance in the pupil plane (as a function of the spatial variable x ) is given by
( ) ()
The incoherent PSF is the magnitude square of the Fourier transform of ( ) x
of wavelengths, and is given by
where u is the spatial frequency variable in the far field approximation, and equivalently the
coordinates in the object plane. The image ( )
is modeled as the convolution of the incoherent
PSF and scene ( )
( )( ) ( )
The goal now is to select
α is such a manner that
over the range of wavelength between min
, integrated over a band
sin Eq 2
approximates the PSF of the bulk prism
Proc. of SPIE Vol. 6978 69780G-4
The problem can be formulated as a minimization of the mean square error (MSE) between the
ideal PSF and
represents the incoherent PSF of the aperture with the bulk prism
phase, we define the MSE metric
( )( )
and numerically minimize its value over all possible choices of
An insight to be gained from Eq. 2 is that the sinc function is due to the small micro-prism only, and
points in the desired look direction. However, its main-lobe is broad and cannot produce the desired
resolution in the image. On the other hand, the summation term is key to recovering resolution, and
so the phases of
α must be chosen such that this term converges to a narrow spot under the
mainlobe of the sinc envelope. The two terms then together form a well behaved PSF as we shall
see in following Section.
. Thus if ( )
3. Preliminary results
The simulations presented in this section were conducted in Matlab using the “fmincon” function to
carry out the optimization posed in Eq. (3). Approximately 400 random sections of a bulk prism
were used (i.e. j=1…400), over a narrow range of 0.2 microns in wavelengths. Figure 5(a) shows
the 2D random aperture pattern. In the absence of any pointing mechanism, the PSF of this random
sparse aperture would exhibit a well formed peak looking straight along the optical axis. Assume
now that a bulk prism with a slope towards the top left is placed across the aperture. The equivalent
PSF is shown in 5(b) which shows that the main lobe of the PSF (a narrow peak) has shifted to the
new look direction.
Figure 5: The random sparse aperture in (a) is combined with a bulk prism with a slope pointing to the
top left. The resulting “ideal” PSF is shown in (b)
Proc. of SPIE Vol. 6978 69780G-5
The result of using only the micro-prisms and setting all the shims to zero (i.e.
is shown in Figure 6(a). Essentially the terms in the summation in Eq. 2 no longer converge to
produce a well defined PSF, and spurious peaks due to diffraction are observed. However after
optimization, the structure of the PSF is greatly improved as shown in Figure 6(b). Careful
examination shows some degradation of the main peak as compared to the ideal bulk prism PSF.
However, the PSF is still sufficiently improved so that well resolved images can be obtained in the
desired look direction.
α is zero for all j)
Figure 6: Prior to optimization, the PSF of the sparse aperture with only micro-prisms and no shims shown in
(a) does not exhibit a peak in the desired look direction. However, after optimization the diffraction effects are
suppressed, and the PSF shows a peak in the right direction. This compensated PSF has the smallest MSE
with respect to the ideal bulk prism PSF shown in Figure 5(b).
Finally, Figure 7 shows the phase values produced by the shims in radians. In this case, the phase
was restricted to a range of ± 6pi. The length of the shims, d, can be calculated using the following
Thus for instance, if the refractive index is n = 3.4, and the nominal wavelength is λ = 4.65
microns, the shims have to be at most 11.6 microns to yield the maximum phase range of ± 6pi.
This is substantially smaller than the size of the bulk prism (which is in the order of millimeters),
and hence much more realizable using existing optical fabrication techniques.
Proc. of SPIE Vol. 6978 69780G-6
+* — -
+ ++ :+
++ + +++ +
++ V+ +
4++ **++ + +++ +
+ + +
+ + *
+ ++:+ ++;++ :++++ +
+ + +
+ ++ +
+ + +++ ++
+ ;+t++s++ +++ *
Figure 7: Optimum phase values after optimization ranging between ± 6pi require shims that are no
more 11.6 microns at a wavelength of 4.65 microns using a material with refractive index of 3.4.
We have presented a technique for interleaving sparse sub-apertures that allow different look
directions to be imaged through the same aperture. This is accomplished using micro-prisms and
phase delays (shims) optimized to produce a well behaved PSF over a band of wavelengths. The
advantage is that need for interleaving bulk prisms is avoided while minimizing the effects of
chromaticity and diffractions.
Future work will be directed at simultaneously optimizing PSFs for many look directions using
much large sparse apertures. The approach needs to be applied to particular system and device
parameters, and performance will be experimentally verified. The results of these activities will be
reported in a future publication.
1. E. R. Dowski, Jr. and W. Thomas Cathey, "Extended depth of field through wave-front
coding," Appl. Opt. 34, 1859- (1995)
2. Pawan K. Baheti and Mark A. Neifeld, “Imaging with PCA-Based Active Illumination”, in
Frontiers of Optics 2005, OSA Annual meeting, Tucson, AZ, USA, 2005.
3. Robert Muise; Abhijit Mahalanobis, “Recent results of integrated sensing and processing
with hyperspectral imager,” SPIE Proceeding Vol. 6574, April 2007
Proc. of SPIE Vol. 6978 69780G-7
4. Goodwin-Johansson, S. H.; Davidson, M. R.; Dausch, D. E.; Holloway, P. H.; McGuire, G, Download full-text
“Reduced voltage artificial eyelid for protection of optical sensors”, Proc. SPIE Vol. 4695,
p. 451-458, 2002.,
5. H.C. Andrews, B.R. Hunt, “Digital Image Restoration,” Englewood Cliffs, Prentice Hall,
Proc. of SPIE Vol. 6978 69780G-8