Michelle Girvan’s research while affiliated with Santa Fe Institute and other places

Reservoir computers (RCs) are powerful machine learning architectures for time series prediction. Recently, next generation reservoir computers (NGRCs) have been introduced, offering distinct advantages over RCs, such as reduced computational expense and lower training data requirements. However, NGRCs have their own practical difficulties, including sensitivity to sampling time and type of nonlinearities in the data. Here, we introduce a hybrid RC-NGRC approach for time series forecasting of dynamical systems. We show that our hybrid approach can produce accurate short-term predictions and capture the long-term statistics of chaotic dynamical systems in situations where the RC and NGRC components alone are insufficient, e.g., due to constraints from limited computational resources, sub-optimal hyperparameters, sparsely sampled training data, etc. Under these conditions, we show for multiple model chaotic systems that the hybrid RC-NGRC method with a small reservoir can achieve prediction performance approaching that of a traditional RC with a much larger reservoir, illustrating that the hybrid approach can offer significant gains in computational efficiency over traditional RCs while simultaneously addressing some of the limitations of NGRCs. Our results suggest that the hybrid RC-NGRC approach may be particularly beneficial in cases when computational efficiency is a high priority and an NGRC alone is not adequate.

While deep neural networks have excelled at static data such as images, temporal data - the data that these networks will need to process in the real world - remains an open challenge. Handling temporal data with neural networks requires one of three options: backpropagation through time using recurrent neural networks (RNNs), treating the time series as static data for a convolutional neural network (CNNs) or attention-based transformer architectures. RNNs are an elegant autoregressive network type that naturally keep a memory of the past while performing computations. Although recurrent networks such as LSTMs have shown strong success across a multitude of fields and tasks such as natural language processing, they can be difficult to train. Transformers and 1-D CNNs, two feed-forward alternatives for temporal data, have gained popularity but in their base forms lack memory to keep track of past activations. Random recurrent networks, also known as Reservoir Computing, have shown that one need not necessarily backpropagate the error through the recurrent component. Here, we propose a novel hybrid approach that brings together the temporal memory capabilities of a random recurrent network with the powerful learning capacity of a deep temporal convolutional readout, which we call t-ConvESN. We experimentally verify that although the recurrent component remains random and unlearned, its combination with a deep readout achieves superior accuracy on a number of datasets from the UCR time series classification dataset collection compared to other state of the art deep learning architectures. Our experiments also show that our proposed method excels in datasets in the low-data regime.

Recent work has shown that machine learning (ML) models can be trained to accurately forecast the dynamics of unknown chaotic dynamical systems. Such ML models can be used to produce both short-term predictions of the state evolution and long-term predictions of the statistical patterns of the dynamics (``climate''). Both of these tasks can be accomplished by employing a feedback loop, whereby the model is trained to predict forward one time step, then the trained model is iterated for multiple time steps with its output used as the input. In the absence of mitigating techniques, however, this technique can result in artificially rapid error growth, leading to inaccurate predictions and/or climate instability. In this article, we systematically examine the technique of adding noise to the ML model input during training as a means to promote stability and improve prediction accuracy. Furthermore, we introduce Linearized Multi-Noise Training (LMNT), a regularization technique that deterministically approximates the effect of many small, independent noise realizations added to the model input during training. Our case study uses reservoir computing, a machine-learning method using recurrent neural networks, to predict the spatiotemporal chaotic Kuramoto-Sivashinsky equation. We find that reservoir computers trained with noise or with LMNT produce climate predictions that appear to be indefinitely stable and have a climate very similar to the true system, while reservoir computers trained without regularization are unstable. Compared with other types of regularization that yield stability in some cases, we find that both short-term and climate predictions from reservoir computers trained with noise or with LMNT are substantially more accurate. Finally, we show that the deterministic aspect of our LMNT regularization facilitates fast hyperparameter tuning when compared to training with noise.

Forecasting the dynamics of large, complex, sparse networks from previous time series data is important in a wide range of contexts. Here we present a machine learning scheme for this task using a parallel architecture that mimics the topology of the network of interest. We demonstrate the utility and scalability of our method implemented using reservoir computing on a chaotic network of oscillators. Two levels of prior knowledge are considered: (i) the network links are known, and (ii) the network links are unknown and inferred via a data-driven approach to approximately optimize prediction.

Echo State Networks (ESN) are a class of recurrent neural networks that can learn to regress on or classify sequential data by keeping the recurrent component random and training only on a set of readout weights, which is of interest to the current edge computing and neuromorphic community. However, they have struggled to perform well with regression and classification tasks and therefore, could not compete in performance with traditional RNNs, such as LSTM and GRU networks. To address this limitation, we have developed a novel hybrid network, called Parallelized Deep Readout Echo State Network (PDR-ESN) that combines the deep learning readout with a fast random recurrent component, with multiple ESNs computing in parallel. We show the PDR-ESN architecture allows for different configurations of the sub-reservoirs, leading to different variants which we explore. Our findings suggest that different variants of the PDR-ESN offer various advantages in different task domains, with some performing better in regression and others in classification. In all cases, our PDR-ESN architecture outperforms the corresponding gradient-based LSTM and GRU architectures in terms of training time as well as accuracy. To further evaluate, we also compared against a Transformer encoder classifier, where the PDR-ESN outperformed on all tasks. We conclude that our proposed network demonstrates a good trade-off between the fast training times of traditional ESNs with the accuracy of deep backpropagation for real-world tasks. We hope that this architecture offers an alternative approach to sequential processing for edge computing as well as more biologically-realistic network development.

The MYC oncogene has been studied for decades, yet there is still intense debate over how this transcription factor controls gene expression. Here, we seek to answer these questions with an in vivo readout of discrete events of gene expression in single cells. We engineered an optogenetic variant of MYC (Pi-MYC) and combined this tool with single-molecule RNA and protein imaging techniques to investigate the role of MYC in modulating transcriptional bursting and transcription factor binding dynamics in human cells. We find that the immediate consequence of MYC overexpression is an increase in the duration rather than in the frequency of bursts, a functional role that is different from the majority of human transcription factors. We further propose that the mechanism by which MYC exerts global effects on the active period of genes is by altering the binding dynamics of transcription factors involved in RNA polymerase II complex assembly and productive elongation.

Recent research has established the effectiveness of machine learning for data-driven prediction of the future evolution of unknown dynamical systems, including chaotic systems. However, these approaches require large amounts of measured time series data from the process to be predicted. When only limited data is available, forecasters are forced to impose significant model structure that may or may not accurately represent the process of interest. In this work, we present a Meta-learning Approach to Reservoir Computing (MARC), a data-driven approach to automatically extract an appropriate model structure from experimentally observed "related" processes that can be used to vastly reduce the amount of data required to successfully train a predictive model. We demonstrate our approach on a simple benchmark problem, where it beats the state of the art meta-learning techniques, as well as a challenging chaotic problem.

The current challenges at the forefront of data-enabled science and engineering require interdisciplinary solutions. Yet most traditional doctoral programs are not structured to support successful interdisciplinary research. Here we describe the design of and students’ experiences in the COMBINE (Computation and Mathematics for Biological Networks) interdisciplinary graduate program at the University of Maryland. COMBINE focuses on the development and application of network science methods to biological systems for students from three primary domains: life sciences, computational/engineering sciences, and mathematical/physical sciences. The program integrates three established models (T-shaped, pi-shaped and shield-shaped) for interdisciplinary training. The program components largely fall into three categories: (1) core coursework that provides content expertise, communication, and technical skills, (2) discipline-bridging elective courses in the two COMBINE domains that complement the student’s home domain, (3) broadening activities such as workshops, symposiums, and formal peer-mentoring groups. Beyond these components, the program builds community through both formal and informal networking and social events. In addition to the interactions with other program participants, students engage with faculty in several ways beyond the conventional adviser framework, such as the requirement to select a second out-of-field advisor, listening to guest speakers, and networking with faculty through workshops. We collected data through post-program surveys, interviews and focus groups with students, alumni and faculty advisors. Overall, COMBINE students and alumni reported feeling that the program components supported their growth in the three program objectives of Network Science & Interdisciplinarity, Communication, and Career Preparation, but also recommended ways to improve the program. The value of the program can be seen not only through the student reports, but also through the students’ research products in network science which include multiple publications and presentations. We believe that COMBINE offers an effective model for integrated interdisciplinary training that can be readily applied in other fields.

Forecasting the dynamics of large complex networks from previous time-series data is important in a wide range of contexts. Here we present a machine learning scheme for this task using a parallel architecture that mimics the topology of the network of interest. We demonstrate the utility and scalability of our method implemented using reservoir computing on a chaotic network of oscillators. Two levels of prior knowledge are considered: (i) the network links are known; and (ii) the network links are unknown and inferred via a data-driven approach to approximately optimize prediction.