Automated Identification of Filaments in Cryoelectron Microscopy Images

Abstract

Since the foundation for the three-dimensional image reconstruction of helical objects from electron micrographs was laid more than 30 years ago, there have been sustained developments in specimen preparation, data acquisition, image analysis, and interpretation of results. However, the boxing of filaments in large numbers of images—one of the critical steps toward the reconstruction at high resolution—is still constrained by manual processing even though interactive interfaces have been built to aid the tedious and sometimes inaccurate boxing process. This article describes an accurate approach for automated detection of filamentous structures in low-contrast images acquired in defocus pairs using cryoelectron microscopy. The performance of the approach has been evaluated across various magnifications and at a series of defocus values using tobacco mosaic virus (TMV) preserved in vitreous ice as a test specimen. By integrating the proposed approach into our automated data acquisition and reconstruction system, we are now able to generate a three-dimensional map of TMV to approximately 10-Å resolution within 24 h of inserting the specimen grid into the microscope.

Full text

Page 1

Automated Identification of
Filaments in Cryo-electron
Microscopy Images



Yuanxin Zhu, Bridget Carragher,
David Kriegman, Ronald A. Milligan,
and Clinton S. Potter


(Submitted to Journal of
Structural Biology)


Date Issued: July 16, 2001


The Beckman Institute
Imaging Technology Group
Technical Report 01-012




Copyright © 2001 Board of Trustees of the University of Illinois

The Beckm an Institute for Advanced Science and Technology
Im aging Technology Group
405 N Mathews
Urbana, IL 61801
techreports@itg.uiuc.edu
http://www.itg.uiuc.edu

Page 2

1


Automated Identification of Filaments in Cryo-electron Microscopy Images

Yuanxin Zhu, Bridget Carragher, #David J. Kriegman, *Ronald A. Milligan, Clinton S. Potter

Beckman Institute, and #Department of Computer Science, University of Illinois at Urbana-
Champaign, Urbana, Illinois 61801
*The Scripps Research Institute, La Jolla, California



Corresponding Author:
Yuanxin Zhu
Beckman Institute for Advanced Science and Technology
405 N. Mathews Avenue
Urbana, Illinois 61801
Tel: 217-265-0875; Fax: 217-244-6219; e-mail: zhu4@uiuc.edu

Page 3

2

Abstract
Since the foundation for the three-dimensional image reconstruction of helical objects from
electron micrographs was laid more than 30 years ago, there have been sustained developments
in specimen preparation, data acquisition, image analysis, and interpretation of results.
However, the boxing of filaments in large numbers of images—one of the critical steps toward
the reconstruction at high resolution—is still constrained by manual processing even though
interactive interfaces have been built to aid the tedious and sometimes inaccurate boxing process.
This article describes an accurate approach for automated detection of filamentous structures in
low-contrast images acquired in defocus pairs using cryo-electron microscopy. The performance
of the approach has been evaluated across various magnifications and at a series of defocus
values using tobacco mosaic virus (TMV) preserved in vitreous ice as a test specimen. By
integrating the proposed approach into our automated data acquisition and reconstruction system,
we are now able to generate a three-dimensional map of TMV to approximately 10Å resolution
within 24 hours of inserting the specimen grid into the microscope.


Keywords: cryo-electron microscopy, automation, three-dimensional reconstruction of helical
objects, perceptual organization, phase correlation

Page 4

3
1. Introduction

An important role of cryo-electron microscopy (cryo-EM) is to provide unique information about
biological structure from the molecular to the cellular level and thus to yield insight into
biological function. As pointed out by Nogales and Grigorieff (2001), the most promising
technique for the structural characterization of large molecular machines is EM and image
reconstruction. One of the principal bottlenecks to cryo-EM is the very large number of images
that must be acquired and processed for the structural analysis. This requirement results from the
necessity for using extremely low doses of electrons (10e/Å2) to image the specimen in order to
avoid beam induced radiation damage. As a result, the acquired images have a very low signal-
to-noise ratio and in order to reconstruct a three-dimensional electron density map from the two-
dimensional projections, a very large number of images must be averaged together. The exact
number of images required to complete a three-dimensional reconstruction of a macromolecule
depends on the size of the macromolecule and the desired resolution. However, it is generally
agreed that in order to interpret a structure to atomic resolution, data from possibly hundreds of
thousands of copies of the macromolecule must contribute to the average (Henderson, 1995).
This in turn requires the collection of thousands to tens of thousands of electron micrographs.
One of the current constraints in the field is the manual data acquisition and analysis methods
that are slow and labor-intensive.

Using helical specimens as a driver, we have been developing an integrated system for cryo-EM
that would allow a three-dimensional electron density map to be automatically generated within
hours of a specimen being inserted in the microscope and with no manual intervention required

Page 5

4
by an operator. This promises to improve both the quality and quantity of data collection and
processing. Towards this goal, we have already automated the acquisition of images from the
electron microscope (Potter et al, 99; Carragher et al., 2000). The system performs all the
operations normally performed by an experienced microscopist including identifying suitable
areas of vitreous ice at low magnification, determining the presence and location of specimen on
the grid, adjusting imaging parameters under low dose conditions and acquiring images at high
magnification to either film or a digital camera. The system can acquire up to 1000 images a day
and the overall performance of the automated system is equivalent to that of an experienced
microscopist.

More than 30 years ago, DeRosier and Moore (1970) laid the foundation for the reconstruction of
three-dimensional images of biological structures with helical symmetry from electron
micrographs. Since then, various research groups have developed a variety of tools to aid the
image processing of helical objects (Unwin and Klug, 1974; Stewart et al., 1981; Wagenknecht
et al., 1981; Egelman, 1986; Milligan and Flicker, 1987; Bluemke et al., 1988; Carragher et al.,
1988; Bremer et al., 1994; Whittaker et al., 1995; Owen et al., 1996). Several interactive
interfaces (for example, the IMGBOX, Owen et al., 1996) have been built for extracting (boxing
out regions of) filaments from large numbers of micrographs where multiple filaments in various
orientation and length may be present. However these are essentially manual methods and
limited both in the time required for the process and their accuracy.

Integration of the automated data acquisition with the three-dimensional reconstruction process
requires an automation of the boxing process, i.e. an automated approach for the identification of

Page 6

5
the helical filaments in two-dimensional projection images. This automation is particularly
promising since the boxing step is often the bottleneck for high-resolution reconstruction
(Nogales et al., 2001). To our best knowledge, no technique for this purpose has been reported
so far. On the other hand, research on automated selection of single particles is very active. A
number of methods for automatically identifying single particle images have been proposed (van
Heel 1982; Frank and Wagenknecht, 1984; Harauz and Fong-Lochovsky, 1989; Lata et al, 1995;
Thuman-Commike and Chiu, 1995; Martin et al, 1997; Ludtke et al, 1999). Most of them are
based on the technique of template matching, namely first generating a rotationally averaged
reference particle image (template) and then locating particles by seeking peaks in the space
formed by cross-correlating the template with the entire image. In order to improve
performance, Ludtke et al (1999) explored using multiple reference templates. Besides its heavy
computational requirement, cross-correlation is sensitive to background noise. As Martin et al
(1997) mentioned, it is commonly agreed that it is very difficult to identify peaks in the cross-
correlation maps computed from low-contrast images acquired using cryo-EM.

This paper presents an accurate approach for automated identification of filamentous structures
in low-dose, low-contrast images acquired in defocus pairs using cryo-EM. Images are normally
acquired in defocus pairs from any chosen specimen area using the Leginon system (Carragher et
al, 2000; Potter et al, 1999). The first image is acquired at very near to focus (NTF) conditions
(e.g., -0.3 ~ -0.15µm, where “-” and “+” indicate under-focus and over-focus respectively) and
the second one at a farther from focus (FFF) conditions (e.g., -3 ~ -2µm), see Fig. 2, for example.
The time interval between the two exposures is less than 20s due to the time required to read out
the digital image from the camera. There are at least two major advantages of using a defocus

Page 7

6
pair of images. First, by combining the two images in the defocus pair, relatively high contrast at
both low and high spatial frequencies can be attained. Second, the moderately strong low-
resolution signals in the FFF images make it possible for us to develop algorithms to identify
filamentous structures automatically. The idea of using a defocus pair of images has been
explored by several other researchers. For example, Fuller (Fuller 1987) and Schrag et al. (1989)
used the “out-of-focus” image for determination of the virus orientation and the “in-focus” image
for the reconstruction. Zhou and Chiu (Zhou and Chiu, 1993; Zhou, 1995) used a focal pair of
images to obtain relatively high contrast at both low and high spatial frequencies. They also
exploited the center and distance information determined for the “far-from-focus” micrograph to
aid particle selection in the “close-to-focus” micrograph.

Our approach begins with a three-level perceptual organization algorithm to detect filaments in
the FFF images, see Fig. 1 for a schematic overview. Because the NTF images are acquired very
close to focus (e.g. -0.15µm), the contrast in such images is extremely low. On the other hand,
the NTF image in a defocus pair covers almost the same specimen area as the FFF image. The
relative distance between filaments within the NTF image should be the same as that in the FFF
image. We therefore circumvent directly identifying filaments in the NTF image and instead
detect filaments in the FFF image which is then aligned using phase correlation techniques with
the NTF image. The filaments in the NTF image are then extracted using the coordinates that are
identified in the FFF image and shifted according to the result of the alignment.

The proposed approach was tested and evaluated by applying it to high magnification
micrographs of tobacco mosaic virus (TMV) preserved in vitreous ice. Performance is assessed

Page 8

7
in terms of the percentages of correctly identified filaments, false alarms (incorrectly identified
areas), and false dismissals (unboxed filaments). The calculation is based on a comparison to the
filaments that would have been selected by an experienced operator. False dismissals do not
constitute a serious type of error as long as there are large numbers of filament images available
and the percentage of falsely dismissing is low. This is consistent with algorithms for identifying
single particles (Martin et al., 1997). False alarms do pose a severe error since they are not
projections of real helical filaments and thus introduce additional error into the reconstruction
process. Therefore, a preferred filament detection approach should produce a high percentage of
correctly identified filaments while maintaining a low percentage of false dismissals and a very
low percentage of false alarms.

2 Materials and Methods

2.1 Image acquisition
Our automated acquisition system (Potter et al., 1999, Carragher et al., 2000) was used to record
high magnification images of helical filaments of TMV preserved in vitreous ice. TMV is a
well-characterized helical virus (Jeng et al., 1989). It is often used as a transmission electron
microscopy (TEM) standard for calibration of magnification and determining resolution and
provides a good test specimen for evaluating a new technique. Defocus pairs of images were
collected using a Philips CM200 TEM equipped with a CCD camera. At the beginning of our
work, images were acquired to a 1K×1K Gatan MSC CCD camera at the magnification of
66,000x. At this magnification, the pixel size is 2.8Å and the accumulated dose for high
magnification image area was about 10e/Å2. An example defocus pair of images acquired in this

Page 9

8
way is shown in Fig. 2a and 2b. The images show projections of TMVs whose structures form
helical filaments 18 nm in diameter and 300 nm in length. Later, we have experimented with
images acquired to a newly installed 2K×2K Tietz CCD camera. With the 2K×2K camera,
defocus pairs of images were acquired at the magnification of either 66,000x or 88,000x. In the
case of 88,000x, the pixel size is 1.5Å and the accumulated dose for high magnification image
area was about 12e/ Å2.

2.2 Introduction to perceptual organization
In humans, perceptual organization is the ability to immediately detect relationships such as
collinearity, parallelism, connectivity, and repetitive patterns among image elements (Lowe,
1985). Researchers in human visual perception, especially the Gestalt psychologists (Koffka,
1935; Kohler, 1947), have long recognized the importance of finding organization in sensory
data. The Gestalt law of organization states the common rules by which our visual system
attempts to group information (Koffka, 1935; Kohler, 1947]). Some of these rules include
proximity, similarity, common fate, continuation and closure. In computer vision, as Mohan and
Nevatia (Mohan and Netvatia, 1992) proposed, perceptual organization takes primitive image
elements and generates representations of feature groupings that encode the structural
interrelationships between the component elements. Many perceptual organization algorithms
have been proposed (Lowe, 1987; Mohan and Nevatia, 1992; Sarkar and Boyer, 1993, Shi and
Malik, 2000), and surveys of these works can be found (Palmer, 1983; Lowe, 1985; Sarkar and
Boyer, 1994; Williams and Thornber, 1996).

2.3 Detecting filaments in FFF images

Page 10

9
We will model the filament detection process in FFF images as a three-level perceptual
organization based on the classificatory structure proposed by Sarkar and Boyer (1994) who
arrange the features to be organized into four categories: signal, primitive, structural, and
assembly. At each level, feature elements that are not grouped into higher-level features are
discarded. Briefly, at the signal level, we use a Canny edge detector (Canny, 1986) to detect
weak filament boundaries. A Hough transform followed by detecting end points of line
segments and merging collinear line segments is developed at the primitive level to organize
discontinuous edges into line segments with a complete description. At the structural level, line
segments are grouped into filamentous structures, areas enclosed by pairs of line segments, by
seeking parallelism and utilizing high-level knowledge. In addition, statistical method is used to
separate two filaments if they are joined together end to end.

Edge detection
For noisy images, it is necessary that at the signal level, edge detection be used to enhance low-
level feature elements by organizing interesting boundary points into edge chains and
suppressing the possible effects of noise on higher-level perceptual organization. Edge detection
is typically a three-step process including noise smoothing, edge enhancement and edge
localization (Trucco and Verri, 1998). We adopt probably the most widely used edge detector in
today’s computer vision community, the Canny edge detector (Canny, 1986), to detect these
weak boundaries. Canny edge detection uses linear filtering with a Gaussian kernel to smooth
noise and then computes the edge strength and direction for each pixel in the smoothed image.
Candidate edge pixels are identified as the pixels that survive a thinning process called non-
maximum suppression, followed by hysteresis thresholding on the thinned edge magnitude

Page 11

10
image using a low threshold and a high threshold. All candidate edge pixels below the lower
threshold are labeled as a non-edge and all pixels above the low threshold, which can be
connected to any pixel above the high threshold through a chain of edge pixels, are labeled as
edge pixels.

Selecting the input parameters of the Canny algorithm is a critical step because the resulting edge
quality varies greatly with the choice of parameters. At the signal level, the input parameters
were selected to optimize the quality of edges for the purpose of higher-level perceptual
organization. As Heath et al. (1997) described, three parameters need to be specified for the
Canny edge detector. The first is the standard deviation of the Gaussian filter specified in pixels.
The second and third parameters are respectively the low and high thresholds required by the
hysteresis thresholding. The best single overall parameter set for the Canny edge detector,
identified by the performance measure method and evaluated on a wide variety of real images
(Heath et al., 1997), does not produce a successful result as applied to images in our case. To
select the input parameters suitable for the highly noisy, low-contrast cryo-EM images in our
case, we first select a range of parameters that samples the space broadly enough but not too
coarsely for each parameter, and then chose the best suitable input parameter by visual
inspection of the resulting edge images. While this selection does not guarantee that optimal
input parameter set was identified, it does generate good results in a timely way. Fig. 3a shows
the edges detected in the image shown in Fig. 2b with a suitable parameter set.

Line detection and end points computing

Page 12

11
At the primitive level, the Hough transform (HT) (Hough, 1962), followed by searching end
points of the line segments and merging collinear line segments, is used to organize noisy and
discontinuous edges into complete line segments. To simplify the computation, the
θ ρ − rather
than slope-intercept parameters are used to parameterize lines (Duda and Hart, 1972). The use of
the HT to detect lines in an image involves the computation of HT for each pixel in the image,
accumulating votes in a two dimensional accumulator array and searching the array for peaks
holding information of potential lines present in the image. The accumulator array becomes
complicated when the image contains multiple lines, with multiple peaks, some of which may
not correspond to lines but are artifacts of noise in the image. Therefore, we iteratively detect
the dominant line in the image, remove its contribution from the accumulator array, and then
repeat to find the dominant of the remaining lines.

The peaks in the accumulator array provide only the shortest distance of the line to origin (ρ )
and the angle that the normal makes with the x-axis (θ ) of the image plane. They do not provide
any information regarding the length, position, or end points of the line segments, which are vital
to the detection of acceptable filaments. Several algorithms have been proposed in the literature
for finding the length and end points of a line from the Hough accumulator array (Yamato et al.,
1990; Atiquzzaman and Akhtar, 1994; Karmat-Sadekar and Ganesan, 1998). The drawback of
these algorithms is that they are computationally intensive. Our integrated system requires that
the computation be performed on the order of seconds to maximize throughput.

We extend an algorithm, due to McLaughlin and Alder (1997), for the computation of the end
points of a line segment directly from the image, based on accurate line parameters detected

Page 13

12
using the HT. Accurate line parameters can be detected using the HT since the edge detection
process has removed most noise. At each iteration of detecting the dominant line, we find the
global maximum in the accumulator array and from this compute the equation (parameters) of
the corresponding line. A simple post-processing step is used to find the beginning and end
points of the line, which involves moving a small window (e.g.
1515×
pixels) along the line and
counting the number of edge pixels in the window. This count will rise above a threshold value
at the beginning of the line and decrease below the threshold at the line end. Collinear line
segments are merged by allowing gaps in the line segment while moving the small window along
the line.

Having found two end points of the line segment, we next tag all pixels lying on the line segment
and remove the contribution of these pixels to the Hough accumulator array. The Hough
accumulator array is now becoming equivalent to that of an image that had not contained the
detected line segment. The above processing is repeated with the new global maximum of the
accumulator array. This continues until the maximum is below a threshold value which, found
from experimentation, is recalculated at each iteration. Edges that are not grouped into line
segments are discarded after primitive-level perceptual organization. Fig. 3b shows an example
of the line segments with acceptable length obtained by primitive-level perceptual organization
where we can see that the end points of those line segments were accurately identified and
collinear line segments were successfully merged.

Detection of filamentous structures

Page 14

13
At the structural level, line segments are actively grouped into filaments by seeking parallelism
and employing knowledge obtained from training data. A filament is the area enclosed by two
line segments with a particular structural relationship. Besides parallelism, the two line
segments should also meet some heuristics: (i) The distance between the two line segments
should be between the specified maximal and minimal values, both of which are determined by
statistical analysis on training data. (ii) The shorter one of the two line segments should be no
less than one third of the length of the longer line segment. (iii) There should be some
overlapped area between the two line segments. The overlapped area can be measured from the
correspondence between the end points of the two line segments. At least one third of either one
should overlap with the other line segment. The area enclosed by two line segments that meet
these requirements is considered as a possible filament. Finding every possible filament requires
that the algorithm search for all possible matches in a neighborhood of each line segment. As a
result, there may be situations where a line segment has multiple matches (filaments). In this
case, the match with the largest overlapped area wins the competition. Therefore, each line
segment can be a boundary of only one filament. Like the ungrouped edges in the primitive-
level, unmatched line segments are discarded.

In selecting filaments from the images that will be passed to the three-dimensional reconstruction
algorithms, more stringent constraints must be used to limit the number of filaments in order for
the reconstruction process to be accurate and efficient. First, only the area enclosed by the
overlapped part of the two line segments are clipped as the detected filament. Secondly, as
described earlier, filaments may be joined end to end along their length (see Fig. 3d, for
example) and must be individually segmented prior to the three-dimensional image

Page 15

14
reconstruction. Thus our algorithm must separate a filament aggregation into multiple segments
if it consists of two or more filaments that are joined together end to end (the longest filament in
the image shown in Fig. 3e, for example). Finally, filaments that are shorter than the minimal
length required for further analysis are discarded.

Separation of end-to-end joins
To determine whether a detected filament actually consists of two or more shorter filaments that
are end-to-end joined together, we have adopted a four-step approach. (i) Rotate the image
around its center so that the inclined filament becomes horizontal, and then cut the filament out
of the rotated image. (ii) Enhance possible end-to-end joins by linearly filtering the filament
image with a Gaussian kernel and then computing the gradient magnitude for each pixel in the
smoothed image. (iii) Sum along the columns of the image to generate a one-dimensional
projection in which the global peak corresponds to the position of a possible end-to-end join.
(iv) Determine the location of the end-to-end joins by thresholding based on the statistics
(maxium, mean, and standard deviation) of the one-dimensional projection. Using this method
the end-to-end join was identified in Fig. 3d and the longest filament as shown in Fig. 3e is
extracted for further analysis.

After examining a set of filament images, we found that the metric defined as () Wm µ−
can be
used to accurately discriminate whether a filament contains end-to-end joins, where m and µ
are respectively the maximum and mean of the one-dimensional projection, and W is the width
of the filament. First, a threshold value is learned from the set of images by visual inspection.
Then a filament is classified as one containing one or more end-to-end joins if the value of the