Content uploaded by Y. Bengio
Author content
All content in this area was uploaded by Y. Bengio on Nov 19, 2014
Content may be subject to copyright.
A preview of the PDF is not available
Learning Deep Architectures for AI
Abstract and Figures
Theoretical results strongly suggest that in order to learn the kind of complicated functions that can repre- sent high-level abstractions (e.g. in vision, language, an d other AI-level tasks), one needs deep architec- tures. Deep architectures are composed of multiple levels of non-linear operations, such as in neural nets with many hidden layers or in complicated propositional formulae re-using many sub-formulae. Searching the parameter space of deep architectures is a difficult opti mization task, but learning algorithms such as those for Deep Belief Networks have recently been proposed to tackle this problem with notable success, beating the state-of-the-art in certain areas. This paper d iscusses the motivations and principles regarding learning algorithms for deep architectures, in particular those exploiting as building blocks unsupervised learning of single-layer models such as Restricted Boltzmann Machines, used to construct deeper models such as Deep Belief Networks.
Figures - uploaded by Y. Bengio
Author content
All figure content in this area was uploaded by Y. Bengio
Content may be subject to copyright.
... Nevertheless, neural networks also present challenges [64]. Training a neural network can be computationally expensive, especially for deep architectures or large datasets [65]. ...
Agent-based modeling (ABM) is a powerful computational approach for studying complex biological and biomedical systems, yet its widespread use remains limited by significant computational demands. As models become increasingly sophisticated, the number of parameters and interactions rises rapidly, exacerbating the so-called curse of dimensionality and making comprehensive parameter exploration and uncertainty analyses computationally prohibitive. Surrogate modeling provides a promising solution by approximating ABM behavior through computationally efficient alternatives, greatly reducing the runtime needed for parameter estimation, sensitivity analysis, and uncertainty quantification. In this review, we examine traditional approaches for performing these tasks directly within ABMs -- providing a baseline for comparison -- and then synthesize recent developments in surrogate-assisted methodologies for biological and biomedical applications. We cover statistical, mechanistic, and machine-learning-based approaches, emphasizing emerging hybrid strategies that integrate mechanistic insights with machine learning to balance interpretability and scalability. Finally, we discuss current challenges and outline directions for future research, including the development of standardized benchmarks to enhance methodological rigor and facilitate the broad adoption of surrogate-assisted ABMs in biology and medicine.
... As per studies in [2], it is noted that extracted data from information may sometimes be distorted or compromised. Various techniques for analyzing images are being explored to enhance our understanding of this health data. ...
Medical image analysis is a very risen area of study and the speed and precision necessary in medical image analysis. Deep learningmay aid in resolving medical image processing issues including labelled datasets by experts to learn effectively.In the medical field, working with limited access to large volumes of labeled data can present significant challenges. Another challenge is the complexity of medical data. Therefore, this study proposed a deep neural network-based model for medical imaging to detect osteoporosis using transfer learning with MobileNetV2. Class weights are used to alleviate class imbalance, and the learning rate schedule improves model adaptability. The model was created in two variants: one with a learning rate schedule and class weights with an accuracy of 96%, and the second model with only a learning rate schedule with an accuracy of 94%. The anticipated experimental results should illustrate the efficiency of the proposed framework for the future designing of deep learning models for predicting bone fracture and speeding up medical data analysis and interpretation.
... Nonconvex optimization problems are ubiquitous in the machine-learning and artificial-intelligence fields, and those arising in training deep neural networks (DNN) (Bengio 2009) are particularly interesting from multiple aspects. In particular, because the nonconvex functions that appear in DNN training have many local optimal solutions, stochastic gradient descent (SGD) (Robbins and Monro 1951) and its variants, which use gradients to update the sequence, may fall into local optimal solutions and may not minimize losses sufficiently. ...
Graduated optimization is a global optimization technique that is used to minimize a multimodal nonconvex function by smoothing the objective function with noise and gradually refining the solution. This paper experimentally evaluates the performance of the explicit graduated optimization algorithm with an optimal noise scheduling derived from a previous study and discusses its limitations. The evaluation uses traditional benchmark functions and empirical loss functions for modern neural network architectures. In addition, this paper extends the implicit graduated optimization algorithm, which is based on the fact that stochastic noise in the optimization process of SGD implicitly smooths the objective function, to SGD with momentum, analyzes its convergence, and demonstrates its effectiveness through experiments on image classification tasks with ResNet architectures.
... This is important in the case of analysing data produced by humans. It is suggested that with larger surveys, the neural model will significantly outperform other methods in terms of classifier accuracy, generalisation and scalability (Bengio 2009). However, the NN requires many parameters, hence, it can be slow to train. ...
Public transport (PT) is crucial for enhancing the quality of life and enabling sustainable urban development. As part of the UK Transport Investment Strategy, increasing PT usage is critical to achieving efficient and sustainable mobility. This paper introduces Machine Learning Influence Flow Analysis (MIFA), a novel framework for identifying the key influencers of PT usage. Using survey data from bus passengers in Southern England, we evaluate machine learning models. Subsequently, MIFA uncovers that easy payments, e-ticketing, and mobile applications can substantially improve the PT service. MIFA’s implementation demonstrates that strength and importance lead to specific insights into how service characteristics impact user decisions. Practical implications include deploying smart ticketing systems and contactless payments to streamline bus usage. Our results suggest that these strategies can enable bus operators to allocate resources more effectively, leading to increased ridership and enhanced user satisfaction.
Keyword spotting (KWS) is a critical component of voice-driven smart-device applications, requiring high accuracy, sensitivity, and responsiveness to deliver optimal user experiences. Given the always-on nature of KWS systems, minimizing computational complexity and power consumption is essential, particularly for battery-powered edge devices with constrained resources. In this paper, we propose a compact and highly efficient convolutional neural network (CNN) for edge-based KWS tasks, using the Google Speech Commands (GSC) V2 dataset for training and evaluation. Our model employs modified MobileNetV2 architecture, optimized via knowledge distillation from an ensemble of high-performing CNN models. Experimental results demonstrate that the proposed model achieves 94.48% accuracy on clean test data and significantly outperforms existing state-of-the-art edge models on challenging noisy test sets, reaching 86.38% accuracy. The proposed CNN maintains this superior performance with only 73.8K parameters and 19.5M floating-point operations (FLOPs)—approximately three times fewer FLOPs and substantially fewer parameters than previously reported edge-focused KWS models. Moreover, when evaluated on a realistic and challenging external Kaggle test set, the proposed model shows excellent generalization with 88.38% accuracy, surpassing baseline depthwise separable CNN (DS-CNN) approaches. Upon practical deployment on a widely used embedded computing platform, our optimized model achieved fast inference times between 11 ms and 14 ms per sample, outperforming existing baseline methods and confirming its suitability for real-time applications. This study highlights the successful integration of model compression techniques, including ensemble learning and knowledge distillation, to achieve breakthrough performance improvements in accuracy, robustness to noise, computational efficiency, and inference speed, thereby advancing the practical deployment of high-performance KWS solutions on resource-constrained edge devices.
In this chapter, we look at a wide range of feature learning architectures and deep learning architectures, which incorporate a range of feature models and classification models. This chapter digs deeper into the background concepts of feature learning and artificial neural networks summarized in the taxonomy of Chap. 9, and complements the local and regional feature descriptor surveys in Chaps. 3, 4, 5, and 6. The architectures in the survey represent significant variations across neural-network approaches, local feature descriptor and classification-based approaches, and ensemble approaches. The architecture taken together as the sum of its parts is apparently more important than individual parts or components of the design, such as the choice of feature descriptor, number of levels in the feature hierarchy, number of features per layer, or the choice of classifier. Good results are being reported across a wide range of architectures.
This paper proposes a new conceptual framework—Emergent Non-Markovianity (ENM)—to describe how memory-like behavior, symbolic identity, and coherent meaning emerge in systems that have no memory, no recursion, and no centralized state. We observe ENM in the genetic code, in transformer-based artificial intelligence, in human consciousness, in the structure of Scripture, and ultimately in the theology of the Logos. These systems are not connected by material similarity, but by a deeper structural truth: they project coherence through constraint, not through storage. From pseudoinverse analysis of the codon-to-amino acid map, to narrative fluency in stateless AI models, to the soul’s continuity under spiritual transformation, ENM emerges as a unifying signature of symbolic recurrence without memory. We argue that ENM provides a bridge between scientific, theological, and philosophical domains—and that it mirrors the pattern of Christ as Logos, the one through whom all things cohere. This is not an abstraction. It is the rediscovery of a Word not only spoken, but woven into the structure of all that lives, thinks, and remembers by becoming.
Keywords: emergent non-Markovianity, Logos, memory without memory, symbolic structure, genetic code, transformer models, consciousness, narrative identity, Christology, theology of constraint, pseudoinverse analysis, AI coherence, spiritual formation, self-reference, symbolic topology, divine structure, pattern recognition, epistemic emergence, non-recursive systems, structure and meaning. A collaboration with GPT-4o. CC4.0.
Buku ini hadir sebagai upaya untuk memberikan pemahaman mendalam mengenai dunia kecerdasan buatan (AI), dengan fokus pada evolusi teknologi yang terus berkembang, dari generatif AI hingga sistem agentik yang semakin kompleks. Dalam buku ini, pembaca akan diajak untuk memahami berbagai konsep dasar, aplikasi, dan tantangan yang dihadapi dalam penerapan AI di dunia nyata.
Kecerdasan buatan telah menjadi bagian integral dari perkembangan teknologi modern, mempengaruhi banyak sektor kehidupan, mulai dari industri hingga kehidupan sehari-hari. Melalui pembahasan yang komprehensif dan sistematis, buku ini diharapkan dapat memberikan kontribusi signifikan bagi para pembaca, baik yang terjun di bidang teknologi, mahasiswa, maupun masyarakat umum yang tertarik untuk memahami lebih jauh mengenai potensi dan peran AI dalam dunia kita.
For many types of machine learning algorithms, one can compute the statistically `optimal' way to select training data. In this paper, we review how optimal data selection techniques have been used with feedforward neural networks. We then show how the same principles may be used to select data for two alternative, statistically-based learning architectures: mixtures of Gaussians and locally weighted regression. While the techniques for neural networks are computationally expensive and approximate, the techniques for mixtures of Gaussians and locally weighted regression are both efficient and accurate. Empirically, we observe that the optimality criterion sharply decreases the number of training examples the learner needs in order to achieve good performance.
Thesupport-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data.High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
