No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Higher-Accuracy Photonic Neural Networks via Duplication Schemes for Noise Reduction
All analog signal processing is fundamentally subject to noise, and this is also the case in next generation implementations of optical neural networks (ONNs). Therefore, we propose the first hardware-based approach to mitigate noise in ONNs. A tree-like and an accordion-like design are constructed from a given NN that one wishes to implement. Both designs have the capability that the resulting ONNs gives outputs close to the desired solution. To establish the latter, we analyze the designs mathematically. Specifically, we investigate a probabilistic framework for the tree-like design that establishes the correctness of the design, i.e. for any feed-forward NN with Lipschitz continuous activation functions, an ONN can be constructed that produces output arbitrarily close to the original. ONNs constructed with the tree-like design thus also inherit the universal approximation property of NNs. For the accordion-like design, we restrict the analysis to NNs with linear activation functions and characterize the ONNs’ output distribution using exact formulas. Finally, we report on numerical experiments with LeNet ONNs that give insight into the number of components required in these designs for certain accuracy gains. The results indicate that adding just a few components and/or adding them only in the first (few) layers in the manner of either design can already be expected to increase the accuracy of ONNs considerably. To illustrate the effect we point to a specific simulation of a LeNet implementation, in which adding one copy of the layers components in each layer reduces the mean-squared error (MSE) by 59.1% for the tree-like design and by 51.5% for the accordion-like design. In this scenario, the gap in accuracy of prediction between the noiseless NN and the ONNs reduces even more: 93.3% for the tree-like design and 80% for the accordion-like design.
As society’s appetite for information continues to grow, so does our need to process this information with increasing speed and versatility. Many believe that the one-size-fits-all solution of digital electronics is becoming a limiting factor in certain areas such as data links, cognitive radio, and ultrafast control. Analog photonic devices have found relatively simple signal processing niches where electronics can no longer provide sufficient speed and reconfigurability. Recently, the landscape for commercially manufacturable photonic chips has been changing rapidly and now promises to achieve economies of scale previously enjoyed solely by microelectronics. By bridging the mathematical prowess of artificial neural networks to the underlying physics of optoelectronic devices, neuromorphic photonics could breach new domains of information processing demanding significant complexity, low cost, and unmatched speed. In this article, we review the progress in neuromorphic photonics, focusing on photonic integrated devices. The challenges and design rules for optoelectronic instantiation of artificial neurons are presented. The proposed photonic architecture revolves around the processing network node composed of two parts: a nonlinear element and a network interface. We then survey excitable lasers in the recent literature as candidates for the nonlinear node and microring-resonator weight banks as the network interface. Finally, we compare metrics between neuromorphic electronics and neuromorphic photonics and discuss potential applications.
The spiking neural network architecture (SpiNNaker) project aims to deliver a massively parallel million-core computer whose interconnect architecture is inspired by the connectivity characteristics of the mammalian brain, and which is suited to the modeling of large-scale spiking neural networks in biological real time. Specifically, the interconnect allows the transmission of a very large number of very small data packets, each conveying explicitly the source, and implicitly the time, of a single neural action potential or “spike.” In this paper, we review the current state of the project, which has already delivered systems with up to 2500 processors, and present the real-time event-driven programming model that supports flexible access to the resources of the machine and has enabled its use by a wide range of collaborators around the world.
In this paper, we describe the design of Neurogrid, a neuromorphic system for simulating large-scale neural models in real time. Neuromorphic systems realize the function of biological neural systems by emulating their structure. Designers of such systems face three major design choices: 1) whether to emulate the four neural elements—axonal arbor, synapse, dendritic tree, and soma—with dedicated or shared electronic circuits; 2) whether to implement these electronic circuits in an analog or digital manner; and 3) whether to interconnect arrays of these silicon neurons with a mesh or a tree network. The choices we made were: 1) we emulated all neural elements except the soma with shared electronic circuits; this choice maximized the number of synaptic connections; 2) we realized all electronic circuits except those for axonal arbors in an analog manner; this choice maximized energy efficiency; and 3) we interconnected neural arrays in a tree network; this choice maximized throughput. These three choices made it possible to simulate a million neurons with billions of synaptic connections in real time—for the first time—using 16 Neurocores integrated on a board that consumes three watts.
The computationally demanding nature of Deep Learning (DL) has
fueled the research on neuromorphics due to their potential to provide high-speed and low energy hardware accelerators. To this end, neuromorphic photonics are increasingly gain attention since they can operate in very high frequencies with very low energy consumption. However, they also introduce new challenges in DL training and deployment. In this paper, we propose a novel training method that is able to compensate for quantization noise, which profoundly exists in photonic hardware due to analog-to-digital (ADC) and digital-to-analog (DAC) conversions, targeting photonic neural networks (PNNs) which employ easily saturated activation functions. The proposed method takes into account quantization during training, leading to significant performance improvements during the inference phase. We conduct evaluation experiments on both image classification and time-series analysis tasks, employing a wide range of existing photonic neuromorphic architectures. The evaluation experiments demonstrate the effectiveness of the proposed method when low-bit resolution photonic architectures are used, as well as its generalization ability.KeywordsPhotonic neural networksNeuromorphic computingNeural network quantization
Research in photonic computing has flourished due to the proliferation of optoelectronic components on photonic integration platforms. Photonic integrated circuits have enabled ultrafast artificial neural networks, providing a framework for a new class of information processing machines. Algorithms running on such hardware have the potential to address the growing demand for machine learning and artificial intelligence in areas such as medical diagnosis, telecommunications, and high-performance and scientific computing. In parallel, the development of neuromorphic electronics has highlighted challenges in that domain, particularly related to processor latency. Neuromorphic photonics offers sub-nanosecond latencies, providing a complementary opportunity to extend the domain of artificial intelligence. Here, we review recent advances in integrated photonic neuromorphic systems, discuss current and future challenges, and outline the advances in science and technology needed to meet those challenges.
Magnetic domain walls are information tokens in both logic and memory devices, and hold particular interest in applications such as neuromorphic accelerators that combine logic in memory. Here, we show that devices based on the electrical manipulation of magnetic domain walls are capable of implementing linear, as well as programmable nonlinear, functions. Unlike other approaches, domain-wall-based devices are ideal for application to both synaptic weight generators and thresholding in deep neural networks. Prototype micrometer-size devices operate with 8 ns current pulses and the energy consumption required for weight modulation is ≤ 16 pJ. Both speed and energy consumption compare favorably to other synaptic nonvolatile devices, with the expected energy dissipation for scaled 20 nm devices close to that of biological neurons.
Photonic neuromorphic computing is raising a growing interest as it promises to provide massive parallelism and low power consumption. In this paper, we demonstrate for the first time a feed-forward neural network via an 8x8 Indium Phosphide cross-connect chip, where up to 8 on-chip weighted addition circuits are co-integrated, based on semiconductor optical amplifier technology. We perform the weight calibration per neuron, resulting in a normalized root mean square error smaller than 0.08 and a best case dynamic range of 27 dB. The 4:1 weighted addition operation is executed on-chip and is part of a neuron whose non-linear function is implemented via software. A three feedback loop optimization procedure is demonstrated to enable an output neuron accuracy improvement of up to 55%. The exploitation of this technology as the neural network is evaluated by implementing a trained 3-layer photonic deep neural network to solve the Iris flower classification problem. Prediction accuracy of 85.8% is achieved, with respect to the 95% accuracy obtained via a computer. A comprehensive analysis of the error evolution in our system reveals that the electrical/optical conversions dominate the error contribution, which suggests that an all optical approach is preferable for future neuromorphic computing hardware design.
In an age overrun with information, the ability to process reams of data has become crucial. The demand for data will continue to grow as smart gadgets multiply and become increasingly integrated into our daily lives. Next-generation industries in artificial intelligence services and high-performance computing are so far supported by microelectronic platforms. These data-intensive enterprises rely on continual improvements in hardware. Their prospects are running up against a stark reality: conventional one-size-fits-all solutions offered by digital electronics can no longer satisfy this need, as Moore's law (exponential hardware scaling), interconnection density, and the von Neumann architecture reach their limits. With its superior speed and reconfigurability, analog photonics can provide some relief to these problems; however, complex applications of analog photonics have remained largely unexplored due to the absence of a robust photonic integration industry. Recently, the landscape for commercially-manufacturable photonic chips has been changing rapidly and now promises to achieve economies of scale previously enjoyed solely by microelectronics. The scientific community has set out to build bridges between the domains of photonic device physics and neural networks, giving rise to the field of \emph{neuromorphic photonics}. This article reviews the recent progress in integrated neuromorphic photonics. We provide an overview of neuromorphic computing, discuss the associated technology (microelectronic and photonic) platforms and compare their metric performance. We discuss photonic neural network approaches and challenges for integrated neuromorphic photonic processors while providing an in-depth description of photonic neurons and a candidate interconnection architecture. We conclude with a future outlook of neuro-inspired photonic processing.
Inspired by the brain’s structure, we have developed an efficient, scalable, and flexible non–von Neumann architecture that
leverages contemporary silicon technology. To demonstrate, we built a 5.4-billion-transistor chip with 4096 neurosynaptic
cores interconnected via an intrachip network that integrates 1 million programmable spiking neurons and 256 million configurable
synapses. Chips can be tiled in two dimensions via an interchip communication interface, seamlessly scaling the architecture
to a cortexlike sheet of arbitrary size. The architecture is well suited to many applications that use complex neural networks
in real time, for example, multiobject detection and classification. With 400-pixel-by-240-pixel video input at 30 frames
per second, the chip consumes 63 milliwatts.
The articles published by the Annals of Eugenics (1925–1954) have been made available online as an historical archive intended for scholarly use. The work of eugenicists was often pervaded by prejudice against racial, ethnic and disabled groups. The online publication of this material for scholarly research purposes is not an endorsement of those views nor a promotion of eugenics in any way.
Universal approximation in dropout neural networks
- manita
Universal approximation in dropout neural networks
- O A Manita
- M A Peletier
- J W Portegies
- J Sanders
- A Senen-Cerda