Kunihiko Fukushima

• Senior Researcher at Fuzzy Logic Systems Institute

Deep convolutional neural networks (deep CNNs) show a large power for robust recognition of visual patterns. The neocognitron, which was first proposed by Fukushima (1979), is recognized as the origin of deep CNNs. Its architecture was suggested by the neurophysiological findings on the visual systems of mammals. It acquires the ability to recogniz...

In many deep neural networks for pattern recognition, the input pattern is classified in the deepest layer based on features extracted through intermediate layers. IntVec (interpolating-vector) is known to be a powerful method for this process of classification. Although the recognition error can be made much smaller by IntVec than by WTA (winner-t...

Deep convolutional neural networks (deep CNN) show a large power for robust recognition of visual patterns. The neocognitron, which was first proposed by Fukushima (1979), is a network classified to this category. Its architecture was suggested by neurophysiological findings on the visual systems of mammals. It acquires the ability to recognize vis...

“Neuroengineering: from Neurosciences to Computations” presents seminal contributions at the intersection between neurosciences, artificial neural networks and computation. Papers range from novel proposals on neural simulations to advances in computational applications where mathematical modeling is widely used.

The neocognitron is a deep (multi-layered) convolutional neural network that can be trained to recognize visual patterns robustly. In the intermediate layers of the neocognitron, local features are extracted from input patterns. In the deepest layer, based on the features extracted in the intermediate layers, input patterns are classified into clas...

We present a new deep neural network architecture, motivated by sparse random matrix theory that uses a low-complexity embedding through a sparse matrix instead of a conventional stacked autoencoder. We regard autoencoders as an information-preserving dimensionality reduction method, similar to random projections in compressed sensing. Thus, exploi...

Structures of neural networks are usually designed by experts to fit target problems. This study proposes a method to automate small network design for a regression problem based on the Add-if-Silent (AiS) function used in the neocognitron. Because the original AiS is designed for image pattern recognition, this study modifies the intermediate func...

Emerging and novel Bioinspired Artificial Neural Networks (BIANN) provided new interdisciplinary approaches for solution of complicated and intractable problems. How can engineering, mathematics, computation, Artificial Intelligence (AI) and Knowledge Engineering (KE) find inspiration in the behavior and internal functioning of physical, biological...

The neocognitron is a multi-layered convolutional network that can be trained to recognize visual patterns robustly. This paper discusses a new neocognitron, which uses the add-if-silent rule for training intermediate layers and the method of interpolating-vector for classifying patterns at the highest stage of the hierarchical network. By the add-...

This paper proposes an improved add-if-silent rule, which is suited for training intermediate layers of a multi-layered convolutional network, such as a neocognitron. By the add-if-silent rule, a new cell is generated if all postsynaptic cells are silent. The generated cell learns the activity of the presynaptic cells in one-shot, and its input con...

Recent advances in machine learning and computer vision have led to the development of several sophisticated learning schemes for object recognition by convolutional networks. One relatively simple learning rule, the Winner-Kill-Loser (WKL), was shown to be efficient at learning higher-order features in the neocognitron model when used in a written...

The neocognitron, which was proposed by Fukushima [44.1], is a neural network model capable of robust visual pattern recognition. It acquires the ability to recognize patterns through learning. The neocognitron is a hierarchical network consisting of many layers of neuron-like cells. Its architecture was originally suggested from neurophysiological...

This paper proposes new learning rules suited for training multi-layered neural networks and applies them to the neocognitron. The neocognitron is a hierarchical multi-layered neural network capable of robust visual pattern recognition. It acquires the ability to recognize visual patterns through learning. For training intermediate layers of the hi...

Recent advances in machine learning and computer vision have led to the development of several sophisticated learning schemes for object recognition by convolutional networks. One relatively simple learning rule, the Winner-Kill-Loser (WKL), was shown to be efficient at learning higher-order features in the neocognitron model when used in a written...

The neocognitron is a neural network model proposed by Fukushima (1980). Its architecture was suggested by neurophysiological findings on the visual systems of mammals. It is a hierarchical multi-layered network. It acquires the ability to robustly recognize visual patterns through learning. Although the neocognitron has a long history, modificatio...

The neocognitron is a hierarchical, multi-layered neural network capable of robust visual pattern recognition. The neocognitron acquires the ability to recognize visual patterns through learning. The winner-kill-loser is a competitive learning rule recently shown to outperform standard winner-take-all learning when used in the neocognitron to perfo...

The neocognitron is a hierarchical multi-layered neural network capable of robust visual pattern recognition. It has been demonstrated that recent versions of the neocognitron exhibit excellent performance for recognizing handwritten digits. When characters are written on a noisy background, however, recognition rate was not always satisfactory. To...

The neocognitron is a hierarchical multi-layered neural network capable of robust visual pattern recognition. It has been demonstrated that recent versions of the neocognitron exhibit excellent performance for recognizing handwritten digits. When characters are written on a noisy background, however, recognition rate was not always satisfactory. Th...

The neocognitron, which was proposed by Fukushima (1980), is a hierarchical multi-layered neural network capable of robust visual pattern recognition. It acquires the ability to recognize patterns through learning. This paper proposes a new rule for competitive learning, named winner-kill-loser, and apply it to the neocognitron. The winner-kill-los...

When some parts of a pattern are occluded by other objects, the visual system can often estimate the shape of occluded contours from visible parts of the contours. This paper proposes a neural network model capable of such function, which is called amodal completion. The model is a hierarchical multi-layered network that has bottom-up and top-down...

When some parts of a pattern are occluded by other objects, the visual system can often estimate the shape of missing portions from visible parts of the contours. This paper proposes a neural network model capable of such function, which is called amodal completion. The model is a hierarchical multi-layered network that has bottom-up and top-down s...

In an earlier paper [Fukushima, K., & Tohyama, K. (2005). Analysis of optic flow: a neural network model. In ICONIP 2005, International Conference on Neural Information Processing (pp. 312-317); Tohyama, K., & Fukushima, K. (2005). Neural network model for extracting 98 optic flow. Neural Networks, 18(5-6), 549-556], we proposed a neural network ex...

The neocognitron is a hierarchical multilayered neural network capable of robust visual pattern recognition. This paper discusses recent advances in the neocognitron, showing several types of neocognitron, to which various improvements and modifications have been made.

This paper proposes the use of interpolating vectors for robust pattern recognition. Labeled reference vectors in a multi-dimensional feature space are first produced by a kind of competitive learning. We then assume a situation where interpolating vectors are densely placed along lines connecting all pairs of reference vectors of the same label. F...

This paper proposes a powerful algorithm for pattern recognition, which uses interpolating vectors for classifying patterns. Labeled reference vectors in a multi-dimensional feature space are first produced by a kind of competitive learning. We then assume a situation where virtual vectors, called interpolating vectors, are densely placed along lin...

This paper proposes a powerful algorithm for pattern recognition, which uses interpolating vectors . Labeled reference vectors in a multi-dimensional feature space are first produced by a kind of competitive learning. We then assume that interpolating vectors are densely placed along line segments connecting all pairs of reference vectors of the sa...

In the visual systems of mammals, visual scenes are analyzed in parallel by sepa-rate channels. Information concerning visual motion is mainly analyzed through the occipito-parietal pathway. In area MST of the pathway, there are cells that respond selectively to spe-cific motions of a large area of the visual field, such as rotation, expansion and...

This paper proposes a neural network model that extracts axes of symmetry from visual patterns. The input patterns can be line drawings, plane figures or gray-scaled natural images taken by CCD cameras. The model is a hierarchical multi-layered network, which consists of a contrast-extracting layer, edge- extracting layers (simple and complex types...

This paper offers an artificial neural network that recognizes and segments a face and its components (e.g., eyes and mouth) from a complex background. The selective attention model (Fukushima, 1987) has been extended to have two channels of different resolutions. The high-resolution channel can analyze input patterns in detail, but usually lacks t...

When we travel in an environment, we have an optic flow on the retina. Neurons in the area MST of macaque monkeys are reported to have a very large receptive field and analyze optic flows on the retina. Many MST-cells respond selectively to rotation, expansion/contraction and planar motion of the optic flow. Many of them show position-invariant res...

This paper shows that the introduction of non-uniform blur is very useful for comparing images, and proposes a neural network model that extracts axes of symmetry from visual patterns. The blurring operation greatly increases robustness against deformations and various kinds of noise, and largely reduces computational cost. Asymmetry between two gr...

This paper proposes a neural network model that has an ability to restore missing portions of partly occluded patterns. It is a multi-layered hierarchical neural network, in which visual information is processed by interaction of bottom-up and top-down signals. Memories of learned patterns are stored in the connections between cells. Occluded parts...

When we travel in an environment, we have an optic flow on the retina. Neurons in the area MST of macaque monkeys are reported to analyze optic flow. The MST cells respond selectively to rotation, expansion/contraction and spiral motion. Previously, it was suggested that vector-field calculus is useful for analyzing optic flow. However, few MST mod...

This paper proposes a neural network model that extracts axes of symmetry from visual patterns. The input patterns can be line drawings, plane figures or gray-scaled natural images taken by CCD cameras. The model is a multi-layered network. It has an input layer, a contrast-extracting layer, edge-extracting layers (an S-cell layer and a C-cell laye...

This paper proposes a new neocognitron that accepts incremental learning, without giving a severe damage to old memories or reducing learning speed. The new neocognitron uses a competitive learning, and the learning of all stages of the hierarchical network progresses simultaneously. To increase the learning speed, conventional neocognitrons of rec...

Even an identical image is perceived differently by human beings depending on the shape of occluding objects. This paper proposes a neural network model that has an ability to recognize and restore partly occluded patterns in a similar way as our perception. It is a multi-layered hierarchical neural network, in which visual information is processed...

Even the identical image is perceived differently by human beings depending on the shape of occluding objects. This paper proposes a neural network model that has an ability to recognize and restore partly occluded patterns in a similar way as our perception. It is a multi-layered hierarchical neural network, in which visual information is processe...

We propose a neural network model for the figure-ground organization based on spatial arrangement of contours such as parallelism, symmetry, and on contour convexity. All of them have been manifested as effective factors for figure-ground organization by psychological studies. In order to measure parallelism and symmetry, spatially separated distan...

Modeling neural networks is a powerful approach to uncover the mechanism of the brain, and the results of the research are ready to use for engineering applications. This paper introduces several models for vision from recent works by the author. (1) Increased recognition rate of the neocognitron: to increase the recognition rate of the neocognitro...

The author previously proposed a neural network model neocognitron for robust visual pattern recognition. This paper proposes an improved version of the neocognitron and demonstrates its ability using a large database of handwritten digits (ETL1).To improve the recognition rate of the neocognitron, several modifications have been applied: such as,...

This paper proposes a neural network model that has an ability to restore the missing portions of partly occluded patterns. It is a multi-layered hierarchical neural network, in which visual information is processed by interaction of bottom-up and top-down signals. Occluded parts of a pattern are reconstructed mainly by feedback signals from the hi...

This paper proposes a new neocognitron that accepts incremental learning, without giving a severe damage to old memories or reducing learning speed. The new neocognitron uses a competitive learning, and the learning of all stages of the hierarchical network progresses simultaneously. To increase the learning speed, conventional neocognitrons of rec...

Proposes a neural network model that has the ability to repair the missing portions of partly occluded patterns. It is a multi-layered hierarchical neural network, in which visual information is processed by interaction of bottom-up and top-down signals. If a partly occluded pattern is unfamiliar to the model, the model tries to reconstruct the ori...

We propose a neural network model of the 2D invariant pattern recognition including a mechanism of saccadic eye movement. The model extracts every spatial relation between two primitive features (feature-relation) from a stimulus. The mechanism of saccadic eye movement enables the network to obtain all feature-relations present in the stimulus. Thi...

Human beings are often able to read a letter or word partly occluded by contaminating ink stains. However, if the stains are completely erased and the occluded areas of the letter are changed to white, we usually have difficulty in reading the letter. In this article I propose a hypothesis explaining why a pattern is easier to recognize when it is...

The author (1988) and Fukushima and Miyake (1982) proposed a neural network model, neocognitron, for robust visual pattern recognition. This paper proposes an improved version of the neocognitron and demonstrates its ability using a large database of handwritten digits (ETL-1). To improve the recognition rate of the neocognitron, several modificati...

To capture and process visual information flexibly and efficiently from changing external world, the function of active and adaptive information processing is indispensable. Visual information processing in the brain can be interpreted as a process of eliminating irrelevant information from a flood of signals received by the retina. Selective atten...

We discuss the properties of equilibrium states in an autoassociative memory model storing hierarchically correlated patterns (hereafter, hierarchical patterns). We will show that symmetric mixed states (hereafter, mixed states) are bistable on the associative memory model storing the hierarchical patterns in a region of the ferromagnetic phase. Th...

We offer a neural network model which has the ability of sift, scale, and deformation-invariant pattern recognition. The model extracts spatial relations between fixating feature and each of peripheral features. The mechanism of saccadic eye movement enables the model to analyze whole pattern-structure. The ability of the model has been tested on a...

This paper proposes a hypothesis explaining why a pattern is easier to be recognized when the occluding objects are visible. A neural network model is constructed based on the hypothesis and is demonstrated that the model responds to occluded patterns in a similar way as human beings. The visual system extracts various visual features from the inpu...

This paper proposes a new learning rule by which cells with shift-invariant receptive fields are self-organized. With this learning rule, cells similar to simple and complex cells in the primary visual cortex are generated in a network. To demonstrate the new learning rule, we simulate a three-layered network that consists of an input layer (or the...

A network is proposed to recognize and extract the face as well as eyes and mouth from an image with complex background. The proposed network is an improved model of the selective attention mechanism based on interaction between two processing systems with different resolutions. The network's performance was evaluated by learning of several frontal...

In binocular vision, interocularly unpaired points are often generated as a result of occlusions. These points can only produce false matches, and have no information about binocular disparities. However, recent psychophysical experiments have suggested that interocularly unpaired points play an important role in human stereo perception. This artic...

This paper proposes a new learning rule by which cells with shift-invariant receptive fields are self-organized. Namely, cells similar to simple and complex cells in the primary visual cortex are generated in a network trained by the new leaning rule. To demonstrate the new learning rule, we simulate a three-layered network that consists of an inpu...

Proposes a new learning rule by which cells with shift-invariant receptive fields are self-organized. With this learning rule, cells similar to simple and complex cells in the primary visual cortex are generated in a network. To demonstrate the new learning rule, we simulate a three-layered network that consists of an input layer (the retina), a la...

An abstract is not available.

According to the results of experiments in neural physiology and psychology, we suggest that the site in the brain where data are stored and the site where recalled data (images of the outside world) appear are different. It is understood that the spatial information image of the outside world is represented in a subject-centered pattern with finit...

We have reported previously that the performance of a neocognitron can be improved by a built-in bend-extracting layer. The conventional bend-extracting layer can detect bend points and end points of lines correctly, but not always crossing points of lines. This paper shows that an introduction of a mechanism of disinhibition can make the bend-extr...

First Page of the Article

We (1988) have reported previously that the performance of a neocognitron can be improved by a built-in bend-extracting layer. The conventional bend-extracting layer can detect bend points and end points of lines correctly, but not always crossing points of lines. This paper discusses that an introduction of a mechanism of disinhibition can make th...

This paper proposes a new learning rule by which cells with shift-invariant receptive fields are self-organized. During the learning, straight lines of various orientations sweep across the input layer. With this learning rule, cells similar to simple and complex cells in the primary visual cortex are generated in a network. To demonstrate the new...

This paper offers a neural network model that can memorize and recall spatial maps. When driving through a place we have been before, we can recall and imagine the scenery that we cannot see yet but shall see soon. Triggered by the newly recalled image, we can also recall other scenery further ahead of us. The model emulates such a chain process of...

This paper offers a neural network model that can memorize and recall spatial maps. This is an improved version of the authors' previous model (1996). When driving through a place we have been before, we can recall and imagine the scenery that we cannot see yet but shall see soon. Triggered by the newly recalled image, we can also recall other scen...

Using a large-scale real-world database-the ETL-1 database of the Electrotechnical Laboratory in Japan-we show that a neocognitron trained by unsupervised learning with a winner-take-all process can recognize handwritten digits with a recognition rate higher than 97%. We use the technique of dual thresholds for feature-extracting S-cells, and highe...

We propose a neural network model of the visual system of the brain which processes different kinds of attributes such as form and motion in parallel. The model has two separate channels: a channel processing form and a channel processing motion. Each channel has both forward and backward connections, and exhibits selective attention. The selective...

When driving through a place we have been before, we can recall and imagine the scenery that we shall see soon. Triggered by the newly recalled image, we can also recall other scenery further ahead of us. This paper offers a neural network model of such a recalling process. The model uses a correlation matrix memory for memorizing and recalling pat...

The neural networks' ability to robustly recognize patterns is influenced by the selectivity of feature-extracting cells in the networks. This selectivity can be controlled by the threshold values of the cells. This paper proposes to use different threshold values for feature-extracting cells in the learning and recognition phases, when an unsuperv...

A neural network model for speech recognition is proposed, based on neurophysiologicalfindings of the auditory system. The first stage of the system is a feature-extracting module that is a model of the auditory pathway between the cochlea and the auditory cortex. The feature-extracting module extracts constant frequency (CF), FM-ascending (FM-A),...

As one of the systems to recognize the cursive handwritten English connected character sequence, Imagawa et al. proposed a system with the selective attention mechanism. The system of Imagawa et al., however, does not have a very high recognition ability. It is a relatively small‐scale system, where the character category to be recognized is compos...

In the neural network for pattern recognition, when the selectivity of the feature-extracting cell is lowered to enhance the generalizing power, a tendency is produced that patterns with similar shapes but belonging to different categories are confused. Because of this property, it has been difficult to en-hance the discriminating power to separate...

The neocognitron is a hierarchical neural network model capable of deformation-resistant pattern recognition. In the hierarchical network of the neocognitron, feature extraction by the S-cells and a blurring operation by the C-cells are repeated. The ability of each S-cell to robustly extract deformed features is created by the blurring operation o...

The neocognitron is a hierarchical neural network model with pattern recognition ability. The intermediate layers of the neocognitron contain various kinds of cells (feature-extracting cells) that extract partial features of the input patterns. The feature-extracting cell has modifiable input connections, with the connecting weights depending on th...

We have modified the original model of selective attention, which was previously proposed by Fukushima, and extended its ability to recognize and segment connected characters in cursive handwriting. Although the original model of selective attention already had the ability to recognize and segment patterns, it did not always work well when too many...

A connected character recognition system using an architecture based on the selective attention model of a neural network is described. The discussion covers pattern recognition, segmentation, repairing imperfect patterns, attention focusing, search control, attention switching, size and position information, and computer simulation. Improvement of...

The function of generalization is indispensable for training artificial neural networks to robustly recognize patterns. The ability to generalize is acquired by placing constraints on the network's architecture. In order to enable an artificial network to emulate the same function of generalization as human beings, it is essential to design the net...

Using a multilayered neural network model to recognize spatio-temporal patterns is proposed. The hierarchical network used in this model consists of three kinds of neuron-like cells: C-cells, which absorb positional errors, D-cells which allow for time distortions, and S-cells which extract specific spatio-temporal features. In the hierarchical net...

A neural network model for speech recognition is proposed, based on neurophysiological findings of the auditory system. The first stage of the system is a feature extracting module. The extracted features are sent to the next stage, the recognition module. The recognition module consists of three blocks of multilayered networks: the upper, middle,...

A model of speech recognition system is proposed and simulated. This model consists of two modules: a feature extraction module which is composed of layers of neuron-like cells with time constants, and a recognition module with delay-lines. This paper offers a detailed explanation of the feature extraction module of this model. The module consists...

The selective attention model proposed by the author is a neural network model which has the ability to segment patterns, as well as the function of recognizing them. The principles of this selective attention model have been extended for the recognition and segmentation of connected characters. The topics discussed include the network architecture...

The recognition of connected characters in cursive handwriting is a difficult task with ordinary pattern-matching techniques, since the shape of the individual character is affected by its preceding and succeeding characters. One of the authors has proposed a neural network model called the selective attention model, which has the ability to recogn...

The neocognitron is a hierarchical neural network model, which is capable of deformation-resistant pattern recognition. In the conventional neocognitron, each cell has a uniform receptive field. In other words, all parts within a receptive field have the same characteristics. Non-uniformity within a receptive field, however, often produces greater...

Modeling neural networks is useful not only in understanding the mechanism of the brain, but also in obtaining new design principles for character recognition systems. With this approach, the author has proposed various models for visual pattern recognition. This paper introduces two of them: the ‘neocognitron’ and ‘selective attention’ models.The...

The authors present an improved neocognitron, in which bend-detecting cells, as well as line-extracting cells, are utilized. In contrast to the conventional neocognitron, which has shown a lesser ability to recognize deformed patterns when trained using unsupervised learning than by using supervised learning, the new system shows considerable robus...