Hyunjune Sebastian Seung

• Massachusetts Institute of Technology

In most complex nervous systems there is a clear anatomical separation between the nerve cord, which contains most of the final motor outputs necessary for behaviour, and the brain. In insects, the neck connective is both a physical and an information bottleneck connecting the brain and the ventral nerve cord (an analogue of the spinal cord) and co...

Understanding the brain requires understanding neurons’ functional responses to the circuit architecture shaping them. Here we introduce the MICrONS functional connectomics dataset with dense calcium imaging of around 75,000 neurons in primary visual cortex (VISp) and higher visual areas (VISrl, VISal and VISlm) in an awake mouse that is viewing na...

Advances in electron microscopy, image segmentation and computational infrastructure have given rise to large-scale and richly annotated connectomic datasets, which are increasingly shared across communities. To enable collaboration, users need to be able to concurrently create annotations and correct errors in the automated segmentation by proofre...

Understanding the relationship between circuit connectivity and function is crucial for uncovering how the brain computes. In mouse primary visual cortex, excitatory neurons with similar response properties are more likely to be synaptically connected1, 2, 3, 4, 5, 6, 7–8; however, broader connectivity rules remain unknown. Here we leverage the mil...

We are in the era of millimetre-scale electron microscopy volumes collected at nanometre resolution1,2. Dense reconstruction of cellular compartments in these electron microscopy volumes has been enabled by recent advances in machine learning3, 4, 5–6. Automated segmentation methods produce exceptionally accurate reconstructions of cells, but post...

Mammalian neocortex contains a highly diverse set of cell types. These cell types have been mapped systematically using a variety of molecular, electrophysiological and morphological approaches1, 2, 3–4. Each modality offers new perspectives on the variation of biological processes underlying cell-type specialization. Cellular-scale electron micros...

Neural circuit function is shaped both by the cell types that comprise the circuit and the connections between them¹. Neural cell types have previously been defined by morphology2,3, electrophysiology⁴, transcriptomic expression5,6, connectivity7, 8–9 or a combination of such modalities10, 11–12. The Patch-seq technique enables the characterization...

Mammalian cortex features a vast diversity of neuronal cell types, each with characteristic anatomical, molecular and functional properties¹. Synaptic connectivity shapes how each cell type participates in the cortical circuit, but mapping connectivity rules at the resolution of distinct cell types remains difficult. Here we used millimetre-scale v...

Neurons in the neocortex exhibit astonishing morphological diversity, which is critical for properly wiring neural circuits and giving neurons their functional properties. However, the organizational principles underlying this morphological diversity remain an open question. Here, we took a data-driven approach using graph-based machine learning me...

As long anticipated (Sandberg and Bostrom 2008; Seung 2012; Szigeti et al. 2014), connectomics is providing a new foundation for brain simulation by replacing theoretical assumptions about network connectivity with solid empirical facts. Connectomics also yields detailed information about neuronal morphology, which is useful for simulating the biop...

A long-standing goal in neuroscience is to understand how a circuit’s form influences its function. Here, we reconstruct and analyze a synaptic wiring diagram of the larval zebrafish brainstem to predict key functional properties and validate them through comparison with physiological data. We identify modules of strongly connected neurons that tur...

As connectomics advances, it will become commonplace to know far more about the structure of a nervous system than about its function. The starting point for many investigations will become neuronal wiring diagrams, which will be interpreted to make theoretical predictions about function. Here I demonstrate this emerging approach with the Drosophil...

Connections between neurons can be mapped by acquiring and analysing electron microscopic brain images. In recent years, this approach has been applied to chunks of brains to reconstruct local connectivity maps that are highly informative 1–6 , but nevertheless inadequate for understanding brain function more globally. Here we present a neuronal wi...

A catalogue of neuronal cell types has often been called a 'parts list' of the brain1, and regarded as a prerequisite for understanding brain function2,3. In the optic lobe of Drosophila, rules of connectivity between cell types have already proven to be essential for understanding fly vision4,5. Here we analyse the fly connectome to complete the l...

The fruit fly Drosophila melanogaster has emerged as a key model organism in neuroscience, in large part due to the concentration of collaboratively generated molecular, genetic and digital resources available for it. Here we complement the approximately 140,000 neuron FlyWire whole-brain connectome1 with a systematic and hierarchical annotation of...

Serial section transmission electron microscopy (TEM) has proven to be one of the leading methods for millimeter-scale 3D imaging of brain tissues at nanoscale resolution. It is important to further improve imaging efficiency to acquire larger and more brain volumes. We report here a threefold increase in the speed of TEM by using a beam deflecting...

In most complex nervous systems there is a clear anatomical separation between the nerve cord, which contains most of the final motor outputs necessary for behaviour, and the brain. In insects, the neck connective is both a physical and information bottleneck connecting the brain and the ventral nerve cord (VNC, spinal cord analogue) and comprises...

High-resolution electron microscopy of nervous systems has enabled the reconstruction of synaptic connectomes. However, we do not know the synaptic sign for each connection (i.e., whether a connection is excitatory or inhibitory), which is implied by the released transmitter. We demonstrate that artificial neural networks can predict transmitter ty...

Normalization is a fundamental operation in image processing. Convolutional nets have evolved over the past decade to include a large number of normalizations, and this architectural shift has proved essential for robust visual artificial intelligence. Here I argue that normalization is the function of a large fraction of neuronal cell types in the...

The reconstruction of neural circuits from serial section electron microscopy (ssEM) images is being accelerated by automatic image segmentation methods. Segmentation accuracy is often limited by the preceding step of aligning 2D section images to create a 3D image stack. Precise and robust alignment in the presence of image artifacts is challengin...

What insects can see has been probed by over a century of behavioral experiments. Motion and color vision have also been studied through neurophysiology in insect brains. Here I study form vision by interpreting a neuronal wiring diagram of the Drosophila optic lobe. The Dm3 "line amacrine" cells are shown to divide into three cell types with orien...

The neocortex is composed of microcircuits built from distinct cell types. Despite significant progress in characterizing these cell types, understanding the full synaptic connections of individual excitatory cells remains elusive. This study investigates the connectivity of arguably the most well recognized excitatory neuron in the neocortex: the...

The neocortex is one of the most critical structures that makes us human, and it is involved in a variety of cognitive functions from perception to sensory integration and motor control. Composed of repeated modules, or microcircuits, the neocortex relies on distinct cell types as its fundamental building blocks. Despite significant progress in cha...

Advances in Electron Microscopy, image segmentation and computational infrastructure have given rise to large-scale and richly annotated connectomic datasets which are increasingly shared across communities. To enable collaboration, users need to be able to concurrently create new annotations and correct errors in the automated segmentation by proo...

Emerging connectomics resources of whole brains consist of large synapse graphs with millions of connections, making analysis of these rich assets challenging. To disseminate them widely we need web-based platforms that allow users to query, visualize, and explore interactively and without requiring advanced programming skills. Here we present Code...

Inverse problems generally require a regularizer or prior for a good solution. A recent trend is to train a convolutional net to denoise images, and use this net as a prior when solving the inverse problem. Several proposals depend on a singular value decomposition of the forward operator, and several others backpropagate through the denoising net...

Connections between neurons can be mapped by acquiring and analyzing electron microscopic (EM) brain images. In recent years, this approach has been applied to chunks of brains to reconstruct local connectivity maps that are highly informative, yet inadequate for understanding brain function more globally. Here, we present the first neuronal wiring...

We present a direct method for limited angle tomographic reconstruction using convolutional networks. The key to our method is to first stretch every tilt view in the direction perpendicular to the tilt axis by the secant of the tilt angle. These stretched views are then fed into a 2-D U-Net which directly outputs the 3-D reconstruction. We train o...

Animal movement is controlled by motor neurons (MNs), which project out of the central nervous system to activate muscles. Because individual muscles may be used in many different behaviors, MN activity must be flexibly coordinated by dedicated premotor circuitry, the organization of which remains largely unknown. Here, we use comprehensive reconst...

Neuronal wiring diagrams reconstructed by electron microscopy1,2,3,4,5 pose new questions about the organization of nervous systems following the time-honored tradition of cross-species comparisons.6,7 The C. elegans connectome has been conceptualized as a sensorimotor circuit that is approximately feedforward,8,9,10,11 starting from sensory neuron...

Neural circuit function is shaped both by the cell types that comprise the circuit and the connections between those cell types1. Neural cell types have previously been defined by morphology 2,3, electrophysiology 4,5, transcriptomic expression 6-8 , connectivity 9-13, or even a combination of such modalities 14-16. More recently, the Patch-seq tec...

Starburst amacrine cells are a prominent neuron type in the mammalian retina that has been well-studied for its role in direction-selective information processing. One specific property of these cells is that their dendrites tightly stratify at specific depths within the inner plexiform layer (IPL), which, together with their unique expression of c...

We are now in the era of millimeter-scale electron microscopy (EM) volumes collected at nanometer resolution. Dense reconstruction of cellular compartments in these EM volumes has been enabled by recent advances in Machine Learning (ML). Automated segmentation methods can now yield exceptionally accurate reconstructions of cells, but despite this a...

To understand how the neocortex underlies our ability to perceive, think, and act, it is important to study the relationship between circuit connectivity and function. Previous research has shown that excitatory neurons in layer 2/3 of the primary visual cortex of mice with similar response properties are more likely to form connections. However, t...

Mammalian cortex features a large diversity of neuronal cell types, each with characteristic anatomical, molecular and functional properties. Synaptic connectivity rules powerfully shape how each cell type participates in the cortical circuit, but comprehensively mapping connectivity at the resolution of distinct cell types remains difficult. Here,...

Neurons in the neocortex exhibit astonishing morphological diversity which is critical for properly wiring neural circuits and giving neurons their functional properties. The extent to which the morphological diversity of excitatory neurons forms a continuum or is built from distinct clusters of cell types remains an open question. Here we took a d...

Like the vertebrate spinal cord, the insect ventral nerve cord (VNC) mediates limb sensation and motor control. Here, we applied automated tools for electron microscopy (EM) volume alignment, neuron reconstruction, and synapse prediction to create a draft connectome of the Drosophila VNC. To interpret the VNC connectome, it is crucial to know its r...

Neurons in the developing brain undergo extensive structural refinement as nascent circuits adopt their mature form. This physical transformation of neurons is facilitated by the engulfment and degradation of axonal branches and synapses by surrounding glial cells, including microglia and astrocytes. However, the small size of phagocytic organelles...

Learning from experience depends at least in part on changes in neuronal connections. We present the largest map of connectivity to date between cortical neurons of a defined type (L2/3 pyramidal cells in mouse primary visual cortex), which was enabled by automated analysis of serial section electron microscopy images with improved handling of imag...

Electron microscopy of biological tissue has recently seen an unprecedented increase in imaging throughput moving the ultrastructural analysis of large tissue blocks such as whole brains into the realm of the feasible. However, homogeneous, high quality electron microscopy staining of large biological samples is still a major challenge. To date, as...

We observed novel classes of cell-cell contacts between retinal starburst amacrine neurons, from finely detailed morphological reconstructions of cells from an electron microscopic image volume of a mouse retina. These contacts have peculiar morphological patterns and traits, different among the respective On and Off starburst amacrine subpopulatio...

Sparse coding has been proposed as a theory of visual cortex and as an unsupervised algorithm for learning representations. We show empirically with the MNIST data set that sparse codes can be very sensitive to image distortions, a behavior that may hinder invariant object recognition. A locally linear analysis suggests that the sensitivity is due...

Stacked unsupervised learning (SUL) seems more biologically plausible than backpropagation, because learning is local to each layer. But SUL has fallen far short of backpropagation in practical applications, undermining the idea that SUL can explain how brains learn. Here we show an SUL algorithm that can perform completely unsupervised clustering...

Sparse coding has been proposed as a theory of visual cortex and as an unsupervised algorithm for learning representations. We show empirically with the MNIST dataset that sparse codes can be very sensitive to image distortions, a behavior that may hinder invariant object recognition. A locally linear analysis suggests that the sensitivity is due t...

Recent works have derived neural networks with online correlation-based learning rules to perform \textit{kernel similarity matching}. These works applied existing linear similarity matching algorithms to nonlinear features generated with random Fourier methods. In this paper attempt to perform kernel similarity matching by directly learning the no...

Benefiting from the rapid development of electron microscopy imaging and deep learning technologies, an increasing number of brain image datasets with segmentation and synapse detection are published. Most of the automated segmentation methods label voxels rather than producing neuron skeletons directly. A further skeletonization step is necessary...

We assembled a semi-automated reconstruction of L2/3 mouse primary visual cortex from ∼250 × 140 × 90 μm³ of electron microscopic images, including pyramidal and non-pyramidal neurons, astrocytes, microglia, oligodendrocytes and precursors, pericytes, vasculature, nuclei, mitochondria, and synapses. Visual responses of a subset of pyramidal cells a...

Due to advances in automated image acquisition and analysis, whole-brain connectomes with 100,000 or more neurons are on the horizon. Proofreading of whole-brain automated reconstructions will require many person-years of effort, due to the huge volumes of data involved. Here we present FlyWire, an online community for proofreading neural circuits...

Inhibitory neurons in mammalian cortex exhibit diverse physiological, morphological, molecular and connectivity signatures. While considerable work has measured the average connectivity of several interneuron classes, there remains a fundamental lack of understanding of the connectivity distribution of distinct inhibitory cell types with synaptic r...

We reconstructed all cell nuclei in a 3D image of a Drosophila brain acquired by serial section electron microscopy (EM). The total number of nuclei is approximately 133,000, at least 87% of which belong to neurons. Neuronal nuclei vary from several hundred down to roughly 5 cubic micrometers. Glial nuclei can be even smaller. The optic lobes conta...

3D electron microscopy (EM) has been successful at mapping invertebrate nervous systems, but the approach has been limited to small chunks of mammalian brains. To scale up to larger volumes, we have built a computational pipeline for processing petascale image datasets acquired by serial section EM, a popular form of 3D EM. The pipeline employs con...

The value of an integrated approach for understanding the neocortex by combining functional characterization of single neuron activity with the underlying circuit architecture has been understood since the dawn of modern neuroscience. However, in practice, anatomical connectivity and physiology have been studied mostly separately. Following in the...

We show dense voxel embeddings learned via deep metric learning can be employed to produce a highly accurate segmentation of neurons from 3D electron microscopy images. A "metric graph" on a set of edges between voxels is constructed from the dense voxel embeddings generated by a convolutional network. Partitioning the metric graph with long-range...

Many approaches to 3D image segmentation are based on hierarchical clustering of supervoxels into image regions. Here we describe a distributed algorithm capable of handling a tremendous number of supervoxels. The algorithm works recursively, the regions are divided into chunks that are processed independently in parallel by multiple workers. At ea...

Electron microscopy of biological tissue has recently seen an unprecedented increase in imaging throughput moving the ultrastructural analysis of large tissue blocks such as whole brains into the realm of the feasible. However, homogeneous, high quality electron microscopy staining of large biological samples is still a major challenge. To date, as...

Neurons in the developing brain undergo extensive structural refinement as nascent circuits adopt their mature form1. This transformation is facilitated by the engulfment and degradation of excess axonal branches and inappropriate synapses by surrounding glial cells, including microglia and astrocytes2,3. However, the small size of phagocytic organ...

Sustained changes in mood or action require persistent changes in neural activity, but it has been difficult to identify the neural circuit mechanisms that underlie persistent activity and contribute to long-lasting changes in behavior. Here, we show that a subset of Doublesex+ pC1 neurons in the Drosophila female brain, called pC1d/e, can drive mi...

Neuronal wiring diagrams reconstructed from electron microscopic images are enabling new ways of attacking neuroscience questions. We address two central issues, modularity and neural coding, by reconstructing and analyzing a wiring diagram from a larval zebrafish brainstem. We identified a recurrently connected “center” within the 3000-node graph,...

We present a semi-automated reconstruction of L2/3 mouse primary visual cortex from 3 million cubic microns of electron microscopic images, including pyramidal and inhibitory neurons, astrocytes, microglia, oligodendrocytes and precursors, pericytes, vasculature, mitochondria, and synapses. Visual responses of a subset of pyramidal cells are includ...

Collision detection is critical for safe robot operation in the presence of humans. Acoustic information originating from collisions between robots and objects provides opportunities for fast collision detection and localization; however, audio information from microphones on robot manipulators needs to be robustly differentiated from motors and ex...

Large scientific projects in genomics and astronomy are influential not because they answer any single question but because they enable investigation of continuously arising new questions from the same data-rich sources. Advances in automated mapping of the brain's synaptic connections (connectomics) suggest that the complicated circuits underlying...

Before backpropagation training, it is common to randomly initialize a neural network so that mean and variance of activity are uniform across neurons. Classically these statistics were defined over an ensemble of random networks. Alternatively, they can be defined over a random sample of inputs to the network. We show analytically and numerically...

The activity and connectivity of inhibitory cells has a profound impact on the operation of neuronal networks. While the average connectivity of many inhibitory cell types has been characterized, we still lack an understanding of how individual interneurons distribute their synapses onto their targets and how heterogeneous the inhibition is onto di...

Convolutional nets have been shown to achieve state-of-the-art accuracy in many biomedical image analysis tasks. Many tasks within biomedical analysis domain involve analyzing volumetric (3D) data acquired by CT, MRI and Microscopy acquisition methods. To deploy convolutional nets in practical working systems, it is important to solve the efficient...

Template matching by normalized cross correlation (NCC) is widely used for finding image correspondences. NCCNet improves the robustness of this algorithm by transforming image features with siamese convolutional nets trained to maximize the contrast between NCC values of true and false matches. The main technical contribution is a weakly supervise...

Learning from experience depends at least in part on changes in neuronal connections. We present the largest map of connectivity to date between cortical neurons of a defined type (L2/3 pyramidal cells), which was enabled by automated analysis of serial section electron microscopy images with improved handling of image defects. We used the map to i...

We show dense voxel embeddings learned via deep metric learning can be employed to produce a highly accurate segmentation of neurons from 3D electron microscopy images. A metric graph on an arbitrary set of short and long-range edges can be constructed from the dense embeddings generated by a convolutional network. Partitioning the metric graph wit...

Multi-order Scaling of High-throughput Transmission Electron Microscopy - Volume 25 Supplement - Christopher S. Own, Theodore DeRego, Lawrence S. Own, Gerward Weppelman, Adrian A. Wanner, Sebastian Ströh, Eric Hammerschmith, Ashwin Vishwanathan, H. Sebastian Seung

Neural circuits can be reconstructed from brain images acquired by serial section electron microscopy. Image analysis has been performed by manual labor for half a century, and efforts at automation date back almost as far. Convolutional nets were first applied to neuronal boundary detection a dozen years ago, and have now achieved impressive accur...

It is now common to process volumetric biomedical images using 3D Convolutional Networks (ConvNets). This can be challenging for the teravoxel and even petavoxel images that are being acquired today by light or electron microscopy. Here we introduce chunkflow, a software framework for distributing ConvNet processing over local and cloud GPUs and CP...

Connectomics aims to recover a complete set of synaptic connections within a dataset imaged by electron microscopy. Most systems for locating synapses use voxelwise classifier models, and train these classifiers to reproduce binary masks of synaptic clefts. However, only recent work has included a way to identify the synaptic partners that communic...

We present a novel method enabling robots to quickly learn to manipulate objects by leveraging a motion planner to generate "expert" training trajectories from a small amount of human-labeled data. In contrast to the traditional sense-plan-act cycle, we propose a deep learning architecture and training regimen called PtPNet that can estimate effect...

We propose a method of aligning a source image to a target image, where the transform is specified by a dense vector field. The two images are encoded as feature hierarchies by siamese convolutional nets. Then a hierarchy of aligner modules computes the transform in a coarse-to-fine recursion. Each module receives as input the transform that was co...

Neural circuits can be reconstructed from brain images acquired by serial section electron microscopy. Image analysis has been performed by manual labor for half a century, and efforts at automation date back almost as far. Convolutional nets were first applied to neuronal boundary detection a dozen years ago, and have now achieved impressive accur...

Convolutional nets have been shown to achieve state-of-the-art accuracy in many biomedical image analysis tasks. Many tasks within biomedical analysis domain involve analyzing volumetric (3D) data acquired by CT, MRI and Microscopy acquisition methods. To deploy convolutional nets in practical working systems, it is important to solve the efficient...

Reconstructing multiple molecularly defined neurons from individual brains and across multiple brain regions can reveal organizational principles of the nervous system. However, high resolution imaging of the whole brain is a technically challenging and slow process. Recently, oblique light sheet microscopy has emerged as a rapid imaging method tha...

Before training a neural net, a classic rule of thumb is to randomly initialize the weights so that the variance of the preactivation is preserved across all layers. This is traditionally interpreted using the total variance due to randomness in both networks (weights) and samples. Alternatively, one can interpret the rule of thumb as preservation...

In the deep metric learning approach to image segmentation, a convolutional net densely generates feature vectors at the pixels of an image. Pairs of feature vectors are trained to be similar or different, depending on whether the corresponding pixels belong to same or different ground truth segments. To segment a new image, the feature vectors are...

Recent developments in serial-section electron microscopy allow the efficient generation of very large image data sets but analyzing such data poses challenges for software tools. Here we introduce Volume Annotation and Segmentation Tool (VAST), a freely available utility program for generating and editing annotations and segmentations of large vol...

When 3D electron microscopy and calcium imaging are used to investigate the structure and function of neural circuits, the resulting datasets pose new challenges of visualization and interpretation. Here, we present a new kind of digital resource that encompasses almost 400 ganglion cells from a single patch of mouse retina. An online "museum" prov...

Most digital brain atlases have macroscopic resolution and are confined to a single imaging modality. Here we present a new kind of resource that combines dense maps of anatomy and physiology at cellular resolution. The resource encompasses almost 400 ganglion cells from a single patch of mouse retina, and a digital “museum” provides a 3D interacti...

Significant advances have been made in recent years on the problem of neural circuit reconstruction from electron microscopic imagery. Improvements in image acquisition, image alignment, and boundary detection have greatly reduced the achievable error rate. In order to make further progress, we argue that automated error detection is essential for...

Neural integrators are involved in a variety of sensorimotor and cognitive behaviors. The oculomotor system contains a simple example, a hindbrain neural circuit that takes velocity signals as inputs and temporally integrates them to control eye position. Here we investigated the structural underpinnings of temporal integration in the larval zebraf...

Convolutional networks (ConvNets), largely running on GPUs, have become the most popular approach to computer vision. Now that CPUs are closing the FLOPS gap with GPUs, efficient CPU algorithms are becoming more important. We propose a novel parallel and vectorized algorithm for N-D convolutional layers. Our goal is to achieve high utilization of a...

For the past decade, convolutional networks have been used for 3D reconstruction of neurons from electron microscopic (EM) brain images. Recent years have seen great improvements in accuracy, as evidenced by submissions to the SNEMI3D benchmark challenge. Here we report the first submission to surpass the estimate of human accuracy provided by the...

State-of-the-art light and electron microscopes are capable of acquiring large image datasets, but quantitatively evaluating the data often involves manually annotating structures of interest. This processis time-consuming and often a major bottleneck in the evaluation pipeline. To overcome this problem, we have introduced the Trainable Weka Segmen...

Convolutional networks (ConvNets) have become a popular approach to computer vision. Here we consider the parallelization of ConvNet training, which is computationally costly. Our novel parallel algorithm is based on decomposition into a set of tasks, most of which are convolutions or FFTs. Theoretical analysis suggests that linear speedup with the...

Once considered provocative, the notion that the wisdom of the crowd is superior to any individual has become itself a piece of crowd wisdom, leading to speculation that online voting may soon put credentialed experts out of business. Recent applications include political and economic forecasting, evaluating nuclear safety, public policy, the quali...