![Reconstruction: Top and side views of Keyhole Lympet Hemocyanin. spatial frequency dependent oscillations into the Fourier space representation of the im- age. The theory of contrast transfer in the electron microscope [5, 6] provides a parametric form for the CTF, the envelope function and the background noise. In our own work [10], we have a completely automated algorithm for estimating the parameters of the contrast transfer function (CTF) of a transmission electron microscope including estimation of the astigmatism. Once estimated, the micrographs can be corrected in order to generate an accurate 3-D reconstruction of the specimen. In [11] we demonstrated how to detect particles in noisy micrographs like the one in Fig. 1 using a cascaded detector based on the face detector of Viola and Jones [14]. The detector is trained on manually selected examples of the particle of interest and randomly selected regions of a micrograph (non-particles). In [16], it was shown to be one of the most effective methods for particle detection in a benchmark dataset of Keyhole Lympet Hemocyanin (KLH). An example of a reconstruction of KLH using the automatically detected particles is shown in Fig. 2. Finally, in [11], we introduced an automated technique for reconstructing a 3-D initial model from multiple electron micrographs without information about orientation of the particle in each image. The method is based on three ideas: First, between any two projections ( i and j ) of a 3-D volume (e.g. a particle), we can compute a constraint on the relative 3-D orientation between the two views. For two images with orthographic viewing directions v i and v j , there is a 3-D direction v i × v j which projects onto each image as directions φ i j and φ ji . These directions can be determined from a projection image (e.g., a transmission electron micrograph, an x-ray, etc.) using a Radon transform. Identifying φ i j and φ ji provides two constraints, called the common lines constraint , on the relative orientation of two views i and j . Second, given three images, there is a common line constraint between each pair of images; for such a triplet, the relative orientations are fully determined in principle up to a reflection ambiguity. Yet because the estimates of φ i j can be noisy and include outliers (mismatches), [9] introduces a robust technique for estimating the 3-D projection orientation by denoising a common lines matrix [ φ i j ] . Finally, once the orientations are determined, the 3-D density is readily estimated using the Fourier Slice Theorem, which states that the 2-D Fourier transform of a projection of a 3-D density is a central slice through the 3-D Fourier transform of the density. Figure 3.a shows two views of a macro-molecule called GroEL produced from a pub- lished 11.5 A ̊ reconstruction [8] as well as the initial model produced with our tech-](https://www.researchgate.net/profile/Serge-Belongie/publication/221260035/figure/fig2/AS:305649234792455@1449883864781/Reconstruction-Top-and-side-views-of-Keyhole-Lympet-Hemocyanin-spatial-frequency_Q320.jpg)
Content uploaded by Serge Belongie
Author content
All content in this area was uploaded by Serge Belongie
Content may be subject to copyright.
Vision in the Small:
Reconstructing the Structure of Protein Macromolecules
from Cryo-Electron Micrographs
Satya Mallick Sameer Agarwal David Kriegman Serge Belongie
Computer Science and Engineering
University of California, San Diego, CA
Bridget Carragher Clinton Potter
National Resource for Automated Molecular Microscopy, and
Department of Cell Biology, The Scripps Research Institute, La Jolla
Abstract
Single particle reconstruction using Cryo-Electron Microscopy (cryo-EM)
is an emerging technique in structural biology for estimating the 3-D struc-
ture (density) of protein macromolecules. Unlike tomography where a large
number of images of a specimen can be acquired, the number of images of
an individual particle is limited because of radiation damage. Instead, the
specimen consists of identical copies of the same protein macro-molecule
embedded in vitreous ice at random and unknown 3-D orientations. Because
the images are extremely noisy, thousands to hundreds-of-thousands of pro-
jections are needed to achieve the desired resolution of 5 ˚
A. Along with dif-
ferences of the imaging modality compared to photographs, single particle
reconstruction provides a unique set of challenges to existing computer vi-
sion algorithms. Here, we introduce the challenge and opportunity of recon-
struction from transmission electron micrographs, and briefly describe our
contributions in areas of particle detection, contrast transfer function (CTF)
estimation, and initial 3-D model construction.
Reconstructing the Structure of Protein Macromolecules
One of the most exciting challenges for biology today is understanding the molecular
machinery of the cell as a working, dynamic system. Critical to this understanding is de-
termining the 3-D structure of protein macromolecules, a task that is often accomplished
using x-ray crystallography. The technique of cryo electron microscopy (cryo-EM) has
a unique role to play in addressing this challenge as it can provide structural informa-
tion of large macromolecular complexes in a variety of conformational and compositional
states while preserved under close to physiological conditions. Traditionally the methods
for cryo-EM have been time consuming and labor intensive, involving data acquisition,
analysis and averaging of thousands to hundreds of thousands of images (views) of the in-
dividual macro-molecular complexes. Thus, over the last few years there has been consid-
erable interest and substantial effort devoted to developing automated methods to improve
the accuracy, robustness, ease of use, and throughput of cryo-EM [1, 2, 3, 12, 15, 17], and
this abstract considers three aspects originally presented in [9, 10, 11].
1
Figure 1: The image on the left shows a typical cryo-EM micrograph containing several
images of a macro-molecule called GroEL. The inset shows a zoomed portion of the
micrograph. The image in the center shows nine projections selected from a micrograph.
Many such projections (∼10,000) are clustered into 50-100 classes. The image on the
right shows the class averages of nine arbitrarily chosen classes. The class averages have
significantly better signal to noise ratio at the expense of finer detail (high resolution
information) contained in the raw projections.
One of the advantages of using cyro-EM is that 3-D structure can be determined with-
out the need for crystallization. It is often very difficult to crystallize large molecules.
Even when crystallization is possible, the structure constrained in crystalline form can
differ from the structure of the macromolecule in its native environment. Cryo-EM there-
fore presents an attractive alternative for structure estimation from a biological point of
view. Though freezing the specimen in vitreous ice preserves it in its close to native state,
the lack of crystalline lattice means that many projections cannot be trivially averaged to
improve the signal to noise ratio.
While the intensity in a photograph is related to the light (radiance) reflected from
surfaces in a scene, the intensity at a point in an image produced in transmission electron
microscopy, like an x-ray, is related to the integral of the scene density along a 3D line
segment between the radiation source and a point on the detector (image plane). Pro-
jection can modelled as orthographic. Computed Tomography (CT) is a technique for
reconstructing the 3D density from a collection of 2D images (aka projections) taken with
a known relation between the radiation source/image plane and the scene. This is akin
to 3D reconstruction from multiple photographs when the camera geometry is known
(multi-view stereo).
In Cryo-EM, a specimen is first frozen in a very thin layer of vitreous (non-crystalline)
ice and then imaged with a transmission electron microscope. To reconstruct protein
macromolecules (aka particle) from the resulting micrographs, CT cannot be used directly
for at least two reasons. First, the energy densities used to acquire a micrograph may
damage the macromolecule, and so only a single usable high resolution image is obtained
per instance of a protein. However, when the macromolecule can be induced to assume
only one (or a small number of) stable confirmations, then multiple copies can be imaged
in turn. Generally, individual particles are frozen into a random and unknown orientation
in the ice; the distribution of orientations may be nonuniformsince some orientations may
be more common. Consequently, to reconstruct 3-D density using CT, the orientation of
each macromolecule must be determined from the image data.
2
The second challenge stems from the low signal to noise ratio. See Fig. 1.a,b for an
example electron micrograph. It is apparent that even detecting and locating particles is
difficult for a human; furthermore it is not reasonable to extract and match local feature
points in images (e.g., corners) and perform structure-from-motion (SFM) using conven-
tional computer vision techniques. Yet, to achieve a desired resolution (<5˚
A) in spite
of the noise, researchers use between 1,000 and 100,000 projections, about two orders of
magnitude more images than typically used in conventional SFM problems.
A single micrograph such as the one in Fig. 1.a contains noisy projections of several
identical randomly oriented particles. The individual particles are selected and cropped
from the micrograph. As can be seen in Fig. 1.b, individual projections are extremely
noisy. The signal to noise ratio can be improved by clustering a large number (∼10,000)
of projections into a few classes (∼50 −100) and averaging within each class; see Fig. 1.c.
Even though averaging leads to smoothing of high resolution information,the detail in the
class averages is sufficient for the purpose of reconstructing an initial model at a resolution
of about 30 −40 ˚
A.
Within the cryo-EM community, a set of techniques for solving the reconstruction
problem have emerged [3], and implementations are available [4, 7, 13]. The process is
essentially the following: First, a rough, usually low resolution and possibly distorted ini-
tial density (initial model) is constructed by some means (e.g., low resolution, higher dose
electron micrographs, x-ray crystallography, single axis or random conical tomography,
known structure of related molecules, assumed structure from other means, etc.). This
model is used to initiate an iterative process where the image plane orientations relative
to the current 3D model are determined (pose estimation), and then the 3D density (a new
model) is reconstructed using CT techniques. The process repeats with this new model.
It should be noted that each iteration may take 12 hours to run, and a full reconstruction
may take a few weeks. In the end, the ability of the iterative process to converge to the
correct solution depends critically on the accuracy of the initial model, and when it does
converge, the number of required iterations also depends upon the accuracy of the initial
model.
The overall processing pipeline can be summarized by the following steps.
1. Specimen preparation.
2. Automated acquisition of electron micrographs.
3. Calibration.
4. Particle detection.
5. Constructing an initial 3-D model.
6. Refinement of the 3-D model.
In our collaborative work between the microscopists and biologists at the Scripps
Research Institute and the computer vision researchers at UCSD, we have addressed a
number of aspects of this processing pipeline. The Leginon system has been developed to
automatically control an electron microscope, select regions of a specimen having suitable
ice thickness, and acquire a large number of micrographs to be used as input [2, 12]
A critical step in the processing and analysis of cryo-EM images involves the esti-
mation of the factors that modulate the image and which must be corrected in order to
generate an accurate 3-D reconstruction of the specimen. Principal among these is the
contrast transfer function (CTF) of the microscope. The effect of the CTF is to introduce
3
Figure 2: Reconstruction: Top and side views of Keyhole Lympet Hemocyanin.
spatial frequency dependent oscillations into the Fourier space representation of the im-
age. The theory of contrast transfer in the electron microscope [5, 6] provides a parametric
form for the CTF, the envelope function and the background noise. In our own work [10],
we have a completely automated algorithm for estimating the parameters of the contrast
transfer function (CTF) of a transmission electron microscope including estimation of the
astigmatism. Once estimated, the micrographs can be corrected in order to generate an
accurate 3-D reconstruction of the specimen.
In [11] we demonstrated how to detect particles in noisy micrographs like the one in
Fig. 1 using a cascaded detector based on the face detector of Viola and Jones [14]. The
detector is trained on manually selected examples of the particle of interest and randomly
selected regions of a micrograph (non-particles). In [16], it was shown to be one of the
most effective methods for particle detection in a benchmark dataset of Keyhole Lympet
Hemocyanin (KLH). An example of a reconstruction of KLH using the automatically
detected particles is shown in Fig. 2.
Finally, in [11], we introduced an automated technique for reconstructing a 3-D initial
model from multiple electron micrographs without information about orientation of the
particle in each image. The method is based on three ideas: First, between any two
projections (iand j) of a 3-D volume (e.g. a particle), we can compute a constraint on
the relative 3-D orientation between the two views. For two images with orthographic
viewing directions viand vj, there is a 3-D direction vi×vjwhich projects onto each
image as directions
φ
i j and
φ
ji . These directions can be determined from a projection
image (e.g., a transmission electron micrograph, an x-ray, etc.) using a Radon transform.
Identifying
φ
i j and
φ
ji provides two constraints, called the common lines constraint, on the
relative orientation of two views iand j. Second, given three images, there is a common
line constraint between each pair of images; for such a triplet, the relative orientations
are fully determined in principle up to a reflection ambiguity. Yet because the estimates
of
φ
i j can be noisy and include outliers (mismatches), [9] introduces a robust technique
for estimating the 3-D projection orientation by denoising a common lines matrix [
φ
i j].
Finally, once the orientations are determined, the 3-D density is readily estimated using
the Fourier Slice Theorem, which states that the 2-D Fourier transform of a projection of
a 3-D density is a central slice through the 3-D Fourier transform of the density.
Figure 3.a shows two views of a macro-molecule called GroEL produced from a pub-
lished 11.5 ˚
A reconstruction [8] as well as the initial model produced with our tech-
4
a.
b.
Figure 3: a. The left column shows the top and side views of a macro-molecule called
GroEL produced from a 11.5 ˚
A reconstruction [8] in a publicly available Molecular Struc-
ture Database. The right column shows the initial model estimated using our method. b.
Starting with the initial model on the left, each column shows two views after each itera-
tion of refinement using routines available in EMAN [7].
nique. 15,839 projections of GroEL were clustered into 40 classes, and the corresponding
40 class averages were generated. A few examples of the class averages are shown in
Fig. 1.c. The initial model was refined using four iterations of standard refinement rou-
tines in EMAN using all 15,839 projections [7]. Fig. 3 shows the progress of refinement
where each iteration takes 9 hours.
Acknowledgments
Part of this work was conducted at the National Resource for Automated Molecular Mi-
croscopy which is supported by NIH through the NCRR P41 program (RR17573). D.
Kriegman and S. Mallick were supported under grant NSF EIA-03-03622, S. Agarwal
was supported under NSF CCF-04-26858, and S. Belongie was supported under NSF
CAREER 0448615, the Alfred P. Sloan Research Fellowship, and DOE under contract
No. W-7405-ENG-48.
5
References
[1] C. R. Booth, W. Jiang, W. Baker, Z. Hong Zhou, S. J. Ludtke, and W. Chiu. A 9 ˚
A single par-
ticle reconstruction from ccd captured images on a 200kv electron cryomicroscope. Journal
of Structural Biology, 147:116–127, 2004.
[2] B. Carragher, N. Kisserberth, D. Kriegman, R. A. Milligan, C. S. Potter, J. Pulokas, and
A. Reilein. LEGINON: An automated system for acquisition of images from vitreous ice
specimens. Journal of Structural Biology, 132:33–45, 2000.
[3] J. Frank. Three-Dimensional Electron Microscopy of Macromolecular Assemblies. Oxford
University Press, 2006.
[4] J. Frank, M. Radermacher, P. Penczek, J. Zhu, et al. SPIDER and WEB: processing and visual-
ization of images in 3D electron microscopy and related fields . J. of Struct. Bio., 116:190–199,
1996.
[5] P. W. Hawkes. The electron microscope as a structure projector. Electron tomography, pages
17–38, 1992.
[6] P. W. Hawkes and E. Kasper. Principles of electron optics: Wave Optics, volume 3. Academic
Press, London, 1994.
[7] S. Ludtke, P. Baldwin, and W. Chiu. EMAN: Semiautomated software for high-resolution
single-particle reconstructions. J. Struc. Bio., 122:82–97, 1999.
[8] S. Ludtke, J. Jakana, J. Song, D. Chuang, and W. Chiu. An 11.5 A Single Particle Reconstruc-
tion of GroEL Using EMAN. J. of Mol. Bio., 314:241–250, 2001.
[9] S. Mallick, S. Agarwal, D. Kriegman, S. Belongie, B. Carragher, and C. Potter. Structure and
View Estimation for Tomographic Reconstruction: A Bayesian Approach. Computer Vision
and Pattern Recognition, 2:2253–2260, 2006.
[10] S. Mallick, D. Kriegman, B. Carragher, and C. Potter. ACE:Automated CTF Estimation.
Ultramicroscopy, 104:8–29, 2005.
[11] S. Mallick, D. Kriegman, and Y. Zhu. Detecting Particles in cryo-EM micrographs using
learned features. Journal of Structural Biology, pages 52–62, January 2004.
[12] C. S. Potter, H. Chu, B. Frey, C. Green, N. Kisserberth, T. J. Madden, K. L. Miller, K. Nahrst-
edt, J. Pulokas, A. Reilein, and et al. Leginon: A system for fully automated acquisition of
1000 micrographs a day. Ultramicroscopy, 77:153–161, 1999.
[13] M. Van Heel. Single-particle electron microscopy: towards atomic resolution. Quart. Rev. of
Bio. phy., 33:307–369, 2000.
[14] Paul Viola and Machael Jones. Robust real-time object detection. Technical Report CRL,
2002.
[15] P. Zhang, A Beatty, J. L. S. Milne, and S. Subramaniam. Automated data collection with tecnai
12 electron microscope: applications for molecular imaging by cryomicroscopy. Journal of
Structural Biology, 135:251–261, 2001.
[16] Y. Zhu, B. Carragher, R. Glaeser, D. Fellmann, C. Bajaj, M Bern, F Mouche, F Haas, R Hall,
D Kriegman, S Ludtke, S Mallick, P Penczek, A Roseman, F Sigworth, N Volkmann, and
C. Potter. Automatic particle selection: results of a comparative study. Journal of Structural
Biology, pages 3–14, 2004.
[17] Y. Zhu, B. Carragher, D. Kriegman, and C. S. Potter. Automated identification of filaments in
cryo-electron microscopy images. Journal of Structural Biology, 135:302–312, 2001.
6
