Mathematical Data Science Archives for Fall 2022 to Spring 2023
Benefits of Weighted Training in Machine Learning and PDE-based Inverse Problems
When: Mon, August 29, 2022 - 10:30amWhere: Online
Speaker: Yunan Yang (Cornell University) - https://yunany.github.io/
Abstract: Many models in machine learning and PDE-based inverse problems exhibit intrinsic spectral properties, which have been used to explain the generalization capacity and the ill-posedness of such problems. In this talk, we discuss weighted training for computational learning and inversion with noisy data. The highlight of the proposed framework is that we allow weighting in both the parameter space and the data space. The weighting scheme encodes both a priori knowledge of the object to be learned and a strategy to weight the contribution of training data in the loss function. We demonstrate that appropriate weighting from prior knowledge can improve the generalization capability of the learned model in both machine learning and PDE-based inverse problems.
Zoom Meeting ID： 927 8056 1489Password：0900Link: https://purdue-edu.zoom.us/j/92780561489?pwd=aXl3cy9Nd1Z5SnJhOW5Id2JDNzRBQT09
Archetypal Analysis
When: Mon, September 12, 2022 - 10:30amWhere: Online
Speaker: Braxton Osting (University of Utah) - https://www.math.utah.edu/~osting/
Abstract: Archetypal Analysis is an unsupervised learning method that uses a convex polytope to summarize multivariate data. For fixed k, the method finds a convex polytope with k vertices, called archetype points, such that the polytope is contained in the convex hull of the data and the mean squared distance between the data and the polytope is minimal. In this talk, I'll give an overview of Archetypal Analysis and discuss our recent results on consistency, a probabilistic method for approximate archetypal analysis, and a version of the problem using the Wasserstein metric. Parts of this work are joint with Katy Craig, Ruijian Han, Dong Wang, Yiming Xu, and Dominique Zosso.
Zoom Meeting ID： 927 8056 1489
Password：0900
Link: https://purdue-edu.zoom.us/j/92780561489?pwd=aXl3cy9Nd1Z5SnJhOW5Id2JDNzRBQT09
Frequency principle and cheap labels in solving PDEs by neural networks
When: Mon, September 19, 2022 - 10:30amWhere: Online
Speaker: Zhiqin Xu (Shanghai Jiao Tong University) - https://ins.sjtu.edu.cn/people/xuzhiqin/
Abstract: In this talk, I would discuss two approaches for solving PDEs by neural networks. The first one is to parameterize the solution by a network. In this approach, neural network suffers from a high-frequency curse, pointed by the frequency principle, i.e., neural network learns data from low to high frequency. To overcome the high-frequency curse, a multi-scale neural network is proposed and verified. The second approach is to express the solution by the form of the Green function and parameterize the Green function by a network. We propose a model-operator-data framework. In this approach, the MOD-Net solves a family of PDEs rather than a specific one and is much more efficient than original neural operator because few expensive labels are required, which are computed on coarse grid points with cheap computation cost and significantly improves the model accuracy.
Zoom Meeting ID： 927 8056 1489
Password：0900
Link: https://purdue-edu.zoom.us/j/92780561489?pwd=aXl3cy9Nd1Z5SnJhOW5Id2JDNzRBQT09
Convergence for score-based generative modeling with polynomial complexity
When: Mon, September 26, 2022 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900 Link: https://go.umd.edu/MTHDataScience
Speaker: Jianfeng Lu (Duke University) - https://services.math.duke.edu/~jianfeng/
Abstract: Score-based generative modeling (SGM) is a highly successful approach for learning a probability distribution from data and generating further samples. We prove the first polynomial convergence guarantees for the core mechanic behind SGM: drawing samples from a probability density p given a score estimate (an estimate of ∇ln p) that is accurate in L2(p). Compared to previous works, we do not incur error that grows exponentially in time or that suffers from a curse of dimensionality. Our guarantee works for any smooth distribution and depends polynomially on its log-Sobolev constant. Using our guarantee, we give a theoretical analysis of score-based generative modeling, which transforms white-noise input into samples from a learned data distribution given score estimates at different noise scales. Our analysis gives theoretical grounding to the observation that an annealed procedure is required in practice to generate good samples, as our proof depends essentially on using annealing to obtain a warm start at each step. Moreover, we show that a predictor-corrector algorithm gives better convergence than using either portion alone. Joint work with Holden Lee and Yixin Tan.
Zoom Meeting ID： 927 8056 1489
Password：0900
Link: https://purdue-edu.zoom.us/j/92780561489?pwd=aXl3cy9Nd1Z5SnJhOW5Id2JDNzRBQT09
Some Perspectives from Inverse Problems on Learning Second-Order Elliptic PDEs from Solution Data
When: Mon, October 3, 2022 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900 Link: https://go.umd.edu/MTHDataScience
Speaker: Kui Ren (Columbia University) - http://www.columbia.edu/~kr2002/
Abstract: In recent years, there have been great interests in discovering structures of partial differential equations from given solution data. Very promising theory and computational algorithms have been proposed for such identification problems in different settings. We will try to review some recent understandings of such PDE learning problems from the perspective of inverse problems. In particularly, we will highlight a few computational and analytical understandings on learning a second-order elliptic PDE from single and multiple solutions.
Zoom Meeting ID： 927 8056 1489
Password：0900
Link: https://purdue-edu.zoom.us/j/92780561489?pwd=aXl3cy9Nd1Z5SnJhOW5Id2JDNzRBQT09
Geometric Variational Autoencoders for Learning Representations of Nonlinear Dynamics and Dimension Reductions
When: Mon, October 17, 2022 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900 Link: https://go.umd.edu/MTHDataScience
Speaker: Paul Atzberger (University of California, Santa Barbara) - http://web.math.ucsb.edu/~atzberg/pmwiki_intranet/index.php?n=AtzbergerHomePage.Homepage?setskin=atzbergerHomepage4
Abstract: We discuss variational autoencoders for learning representations for non-linear dynamics based on manifold latent spaces incorporating prior knowledge of geometric and topological structure. These are referred to as Geometric Dynamic (GD)-VAEs. We show GD-VAEs using manifold latent spaces allow for reducing the number of needed latent dimensions and facilitate learning more interpretable and robust representations of dynamics for making long-time predictions. We discuss challenges and present results using GD-VAEs for learning parsimonious representations for non-linear dynamics of burgers equation, constrained mechanical systems, and dynamics of reaction-diffusion systems. We also make comparisons with other methods, including analytic reduction techniques, Dynamical Model Decomposition (DMD), Proper Orthogonal Decomposition (POD), and standard autoencoders (non-variational). The results indicate some of the ways that non-linear approximation combined with manifold latent spaces can be used to significantly enhance
Zoom Meeting ID： 927 8056 1489
Password：0900
Link: https://purdue-edu.zoom.us/j/92780561489?pwd=aXl3cy9Nd1Z5SnJhOW5Id2JDNzRBQT09
Analysis of adversarial robustness and of other problems in modern machine learning
When: Mon, October 24, 2022 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900 Link: https://go.umd.edu/MTHDataScience
Speaker: Nicolas Trillos (University of Wisconsin–Madison) - https://www.nicolasgarciat.com/
Abstract: Modern machine learning methods, in particular deep learning approaches, have enjoyed unparalleled success in a variety of challenging application fields like image recognition, medical image reconstruction, and natural language processing. While a vast majority of previous research in machine learning mainly focused on constructing and understanding models with high predictive power, consensus has emerged that other properties like stability and robustness of models are of equal importance and in many applications essential. This has motivated researchers to investigate the problem of adversarial training (or how to make models robust to adversarial attacks), but despite the development of several computational strategies for adversarial training and some theoretical development in the broader distributionally robust optimization literature, there are still several theoretical questions about it that remain relatively unexplored. In this talk, I will take an analytical perspective on the adversarial robustness problem and explore three questions: 1)What is the connection between adversarial robustness and inverse problems?, 2) Can we use analytical tools to find lower bounds for adversarial robustness problems?, 3) How do we use modern tools from analysis and geometry to solve adversarial robustness problems? At its heart, this talk is an invitation to view adversarial machine learning through the lens of mathematical analysis, showcasing a variety of rich connections with perimeter minimization problems, optimal transport, mean field PDEs of interacting particle systems, and min-max games in spaces of measures. The talk is based on joint works with Leon Bungert (Bonn), Camilo A. García Trillos (UAL), Matt Jacobs (Purdue), Jakwang Kim (Wisc), and Ryan Murray (NCState).
Zoom Meeting ID： 927 8056 1489
Password：0900
Link: https://purdue-edu.zoom.us/j/92780561489?pwd=aXl3cy9Nd1Z5SnJhOW5Id2JDNzRBQT09
Convergence and optimality of single-layer neural networks for reinforcement learning
When: Mon, October 31, 2022 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900 Link: https://go.umd.edu/MTHDataScience
Speaker: Andrea Agazzi (University of Pisa) - https://services.math.duke.edu/~agazzi/index.html
Abstract: Over the past few years, groundbreaking results have established a convergence theory for wide neural networks in the supervised learning setting. Depending on the scaling of parameters at initialization, the (stochastic) gradient descent dynamics of these models converge towards different deterministic limits known as the mean-field and the lazy training regimes.In this talk, we extend some of these results to examples of prototypical algorithms in reinforcement learning: Temporal-Difference (TD) learning and Policy Gradients. In the first case, we prove convergence and optimality of wide neural network training dynamics in the lazy and mean-field regime, respectively. To establish these results, we bypass the lack of gradient structure of the TD learning dynamics by leveraging Lyapunov function techniques in the lazy training regime and sufficient expressivity of the activation function in the mean-field framework. We further show that similar optimality results hold for wide, single layer neural networkstrained by entropy-regularized softmax Policy Gradients despite the nonlinear and nonconvex nature of the risk function in this setting.This is joint work with Jianfeng Lu.
Zoom Meeting ID： 927 8056 1489
Password：0900
Link: https://purdue-edu.zoom.us/j/92780561489?pwd=aXl3cy9Nd1Z5SnJhOW5Id2JDNzRBQT09
Tensor-network approach for statistical mechanics problem
When: Mon, November 7, 2022 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900 Link: https://go.umd.edu/MTHDataScience
Speaker: Yuehaw Khoo (University of Chicago) - https://www.stat.uchicago.edu/~ykhoo/
Abstract: We develop tensor-network approaches for solving high-dimensional partial differential equations with the goal of characterizing the transition between two states in a statistical mechanics system with high-accuracy. For this purpose we also develop novel generative modeling techniques based on tensor-networks. The proposed method is completely linear algebra based and does not require any optimization.
Zoom Meeting ID： 927 8056 1489
Password：0900
Link: https://purdue-edu.zoom.us/j/92780561489?pwd=aXl3cy9Nd1Z5SnJhOW5Id2JDNzRBQT09
The future of governing equations
When: Mon, November 14, 2022 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900 Link: https://go.umd.edu/MTHDataScience
Speaker: Nathan Kutz (University of Washington) - https://amath.washington.edu/people/j-nathan-kutz
Abstract: A major challenge in the study of dynamical systems is that of model discovery: turning data into reduced order models that are not just predictive, but provide insight into the nature of the underlying dynamical system that generated the data. We introduce a number of data-driven strategies for discovering nonlinear multiscale dynamical systems and their embeddings from data. We consider two canonical cases: (i) systems for which we have full measurements of the governing variables, and (ii) systems for which we have incomplete measurements. For systems with full state measurements, we show that the recent sparse identification of nonlinear dynamical systems (SINDy) method can discover governing equations with relatively little data and introduce a sampling method that allows SINDy to scale efficiently to problems with multiple time scales, noise and parametric dependencies. For systems with incomplete observations, we show that the Hankel alternative view of Koopman (HAVOK) method, based on time-delay embedding coordinates and the dynamic mode decomposition, can be used to obtain a linear models and Koopman invariant measurement systems that nearly perfectly captures the dynamics of nonlinear quasiperiodic systems. Neural networks are used in targeted ways to aid in the model reduction process. Together, these approaches provide a suite of mathematical strategies for reducing the data required to discover and model nonlinear multiscale systems.Bio: Nathan Kutz is the Yasuko Endo and Robert Bolles Professor of Applied Mathematics and Electrical and Computer Engineering at the University of Washington, having served as chair of applied mathematics from 2007-2015. He received the BS degree in physics and mathematics from the University of Washington in 1990 and the Phd in applied mathematics from Northwestern University in 1994. He was a postdoc in the applied and computational mathematics program at Princeton University before taking his faculty position. He has a wide range of interests, including neuroscience to fluid dynamics where he integrates machine learning with dynamical systems and control.
Zoom Meeting ID： 927 8056 1489
Password：0900
Link: https://purdue-edu.zoom.us/j/92780561489?pwd=aXl3cy9Nd1Z5SnJhOW5Id2JDNzRBQT09
Collective variables in complex systems: from molecular dynamics to agent-based models and fluid dynamics
When: Mon, November 21, 2022 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900 Link: https://go.umd.edu/MTHDataScience
Speaker: Peter Koltai (Freie Universität Berlin) - http://userpage.fu-berlin.de/peterkoltai/index.html
Abstract: The identification of persistent forecastable structures in complicated or high-dimensional dynamics is vital for a robust prediction (or manipulation) of such systems in a potentially sparse-data setting. Such structures can be intimately related to so-called collective variables known for instance from statistical physics. We have recently developed a first data-driven technique to find provably good collective variables in molecular systems. Here we will discuss how the concept generalizes to other applications as well, such as fluid dynamics and social or epidemic dynamics.
Zoom Meeting ID： 927 8056 1489
Password：0900
Link: https://purdue-edu.zoom.us/j/92780561489?pwd=aXl3cy9Nd1Z5SnJhOW5Id2JDNzRBQT09
Canceled
When: Mon, November 28, 2022 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900 Link: https://go.umd.edu/MTHDataScience
Canceled
TBA
When: Mon, December 5, 2022 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900 Link: https://go.umd.edu/MTHDataScience
Speaker: George Karniadakis (Brown University) - https://www.brown.edu/research/projects/crunch/george-karniadakis
Abstract: TBA
Zoom Meeting ID： 927 8056 1489
Password：0900
Link: https://purdue-edu.zoom.us/j/92780561489?pwd=aXl3cy9Nd1Z5SnJhOW5Id2JDNzRBQT09
The Mean-Field Ensemble Kalman Filter
When: Mon, January 23, 2023 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Andrew Stuart (Caltech) - http://stuart.caltech.edu/
Abstract: Ensemble Kalman filters constitute a methodology for incorporating noisy data into complex dynamical models to enhance predictive capability. They are widely adopted in the geophysical sciences, underpinning weather forecasting for example, and are starting to be used throughout the sciences and engineering; furthermore, they have been adapted to function as a general-purpose tool for parametric inference. The strength of these methods stems from their ability to operate using complex models as a black box, together with their natural adaptation to high performance computers. In this work we provide, for the first time, theory to elucidate conditions under which this widely adopted methodology provides accurate model predictions and uncertainties for discrete time filtering. The theory rests on a mean-field formulation of the methodology and an error analysis controlling differences between probability measure propagation under the mean-field model and under the true filtering distribution.
The mean-field formulation is based on joint work with Edoardo Calvello (Caltech) and Sebastian Reich (Potsdam).
The error analysis is based on joint work with Jose Carrillo (Oxford), Franca Hoffmann (Caltech) and Urbain Vaes (Paris).
TNet: A Tikhonov Neural Network Approach to Inverse Problems and UQ
When: Mon, January 30, 2023 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Tan Bui-Thanh (The University of Texas at Austin) - https://users.oden.utexas.edu/~tanbui/
Abstract: Deep Learning (DL) by design is purely data-driven and in general does not require physics. This is the strength of DL but also one of its key limitations when applied to science and engineering problems in which underlying physical properties (such as stability, conservation, and positivity) and desired accuracy need to be achieved. DL methods in their original forms are not capable of respecting the underlying mathematical models or achieving desired accuracy even in big-data regimes. On the other hand, many data-driven science and engineering problems, such as inverse problems, typically have limited experimental or observational data, and DL would overfit the data in this case. Leveraging information encoded in the underlying mathematical models, we argue, not only compensates missing information in low data regimes but also provides opportunities to equip DL methods with the underlying physics and hence obtaining higher accuracy. This talk introduces a Tikhonov Network (TNet) that is capable of learning Tikhonov regularized inverse problems. We present and provide intuitions for our formulations for general nonlinear problems. We rigorously show that our TNet approach can learn information encoded in the underlying mathematical models, and thus can produce consistent or equivalent inverse solutions, while naive purely data-based counterparts cannot. Furthermore, we theoretically study the error estimate between TNet and Tikhhonov inverse solutions and under which conditions they are the same. Extension to statistical inverse problems will also be presented.
The Effects of Activation Functions on the Over-smoothing Issue of Graph Convolutional Networks
When: Mon, February 6, 2023 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Bao Wang (The University of Utah) - https://www.sci.utah.edu/people/bwang.html
Abstract: Smoothness has been shown to be crucial for the success of graph convolutional networks (GCNs); however, over-smoothing has become inevitable. In this talk, I will present a geometric characterization of how activation functions of a graph convolution layer affect the smoothness of their input leveraging the distance of graph node features to the eigenspace of the largest eigenvalue of the (augmented) normalized adjacency matrix, denoted as $\gM$. In particular, we show that 1) the input and output of ReLU or leaky ReLU activation function are related by a high-dimensional ball, 2) activation functions can increase, decrease, or preserve the smoothness of node features, and 3) adjusting the component of the input in the eigenspace $\gM$ can control the smoothness of the output of activation functions. Informed by our theory, we propose a universal smooth control term to modulate the smoothness of learned node features and improve the performance of existing graph neural networks.
Manifold Learning between Mathematics and the Sciences
When: Mon, February 13, 2023 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Marina Meila (University of Washington) - https://sites.stat.washington.edu/mmp/
Abstract: Non-linear dimension reduction algorithms can recover the underlying low-dimensional parametrization of high-dimensional point clouds. This area at the frontier of machine learning, statistics, computer science and mathematics is known as Manifold Learning. This talk will extend Manifold Learning in two directions. First, we ask if it is possible, in the case of scientific data where quantitative prior knowledge is abundant, to explain a data manifold by new coordinates, chosen from a set of scientifically meaningful functions? Second, we ask how popular Manifold Learning tools and their applications can be recreated in the space of vector fields and flows on a manifold. For this, we need to transport advanced differential geometric and topological concepts into a data-driven framework. Central to this approach is the order 1-Laplacian of a manifold, $\Delta_1$, whose eigen-decomposition into gradient, harmonic, and curl, known as the Helmholtz-Hodge Decomposition, provides a basis for all vector fields on a manifold. We present an estimator for $\Delta_1$, and based on it we develop a variety of applications. Among them, visualization of the principal harmonic, gradient or curl flows on a manifold, smoothing and semi-supervised learning of vector fields, 1-Laplacian regularization. In topological data analysis, we describe the 1st-order analogue of spectral clustering, which amounts to prime manifold decomposition. Furthermore, from this decomposition a new algorithm for finding shortest independent loops follows. The algorithms are illustrated on a variety of real data sets. Joint work with Yu-Chia Chen, Samson Koelle, Weicheng Wu, Hanyu Zhang and Ioannis Kevrekidis
Self-supervised learning of PDE solution manifolds in function spaces
When: Mon, February 20, 2023 - 10:30amWhere: Online with Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Paris Perdikaris (University of Pennsylvania ) - https://www.amcs.upenn.edu/people/paris-perdikaris
Abstract: While the great success of modern deep learning lies in its ability to approximate maps between finite-dimensional vector spaces, many tasks in science and engineering involve continuous measurements that are functional in nature. For example, in climate modeling one might wish to predict the pressure field over the earth from measurements of the surface air temperature field. The goal is then to learn an operator, between the space of temperature functions to the space of pressure functions. In recent years operator learning techniques using deep neural networks have emerged as a powerful tool for regression problems in infinite-dimensional function spaces. In this talk we present a general approximation framework for neural operators and demonstrate that their performance fundamentally depends on their ability to learn low-dimensional parameterizations of solution manifolds. This motivates new architectures which are able to capture intrinsic low-dimensional structure in the space of target output functions. Additionally, we provide a way to train these models in a self-supervised manner, even in the absence of paired labeled examples. These contributions result in neural PDE solvers which yield fast and discretization invariant predictions of spatio-temporal fields up to three orders of magnitude faster compared to classical numerical solvers. We will also discuss key open questions related to generalization, accuracy, data-efficiency and inductive bias, the resolution of which will be critical for the success of AI in science and engineering.
No equations, no variables, no space, no time: Old and new results on data and the modeling of complex systems
When: Mon, February 27, 2023 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Yannis Kevrekidis (Johns Hopkins University) - https://engineering.jhu.edu/faculty/ioannis-kevrekidis/
Abstract: I will start by showing how several successful NN architectures (ResNets, recurrent nets, convolutional nets, autoencoders, neural ODEs, operator learning....) had been used for nonlinear dynamical system identification (learning ODEs and PDEs) since the early 1990s. Obtaining predictive dynamical equations from data lies at the heart of science and engineering modeling and is the linchpin of our technology. In mathematical modeling one typically progresses from observations of the world (and some serious thinking!) first to equations for a model, and then to the analysis of the model to make predictions. Good mathematical models give good predictions (and inaccurate ones do not) - but the computational tools for analyzing them are the same: algorithms that are typically based on closed-form equations.While the skeleton of the process remains the same, today we witness the development of mathematical techniques that operate directly on observations -data-, and appear to circumvent the serious thinking that goes into selecting variables and parameters and deriving accurate equations. The process then may appear to the user a little like making predictions by "looking in a crystal ball". Yet the "serious thinking" is still there and uses the same -and some new- mathematics: it goes into building algorithms that jump directly from data to the analysis of the model (which is now not available in closed form) to make predictions. Our work here presents a couple of efforts that illustrate this ``new” path from data to predictions. It really is the same old path, but it is traveled by new means.Bio: Yannis Kevrekidis studied Chemical Engineering at the National Technical University in Athens. He then followed the steps of many alumni of that department to the University of Minnesota, where he studied with Rutherford Aris and Lanny Schmidt (as well as Don Aronson and Dick McGehee in Math). He was a Director's Fellow at the Center for Nonlinear Studies in Los Alamos in 1985-86 (when Soviets still existed and research funds were plentiful). He then had the good fortune of joining the faculty at Princeton, where he taught Chemical Engineering and also Applied and Computational Mathematics for 31 years; five years ago he became Emeritus and started fresh at Johns Hopkins (where he somehow is also Professor of Urology). His work always had to do with nonlinear dynamics (from instabilities and bifurcation algorithms to spatiotemporal patterns to data science in the 90s, nonlinear identification, multiscale modeling, and back to data science/ML); and he had the additional good fortune to work with several truly talented experimentalists, like G. Ertl's group in Berlin. When young and promising, he was a Packard Fellow, a Presidential Young Investigator and the Ulam Scholar at Los Alamos National Laboratory. He holds the Colburn, CAST and Wilhelm Awards of the AIChE, the Crawford and the Reid Prizes of SIAM, he is a member of the NAE, the American Academy of Arts and Sciences, and the Academy of Athens.
Learning Operators
When: Mon, March 6, 2023 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Siddhartha Mishra (ETH Zurich) - https://camlab.ethz.ch/the-group/people.html?u=smishra
Abstract: Operators are mapping between infinite-dimensional spaces and arise in a variety of contexts, particularly in the solution of PDEs. The main aim of this lecture would be to introduce the audience to the rapidly emerging area of operator learning, i.e., machine learning operators from data. To this end, we will summarize existing architectures such as DeepONets and Fourier neural operators (FNOs) as well as describe the newly proposed Convolutional Neural Operators (CNOs). Theoretical error estimates for different operator learning architectures will be mentioned and numerical experiments comparing them described. Several open issues regarding operator learning will also be covered. If time permits, we will describe Neural Inverse operators (NIOs): a machine-learning architecture for the efficient learning of a class of inverse problems associated with PDEs.
The mathematics of neural networks: recent advances, thoughts, and a path forward
When: Mon, March 13, 2023 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Mikhail Belkin (University of California, San Diego) - http://misha.belkin-wang.org/
Abstract: The recent remarkable practical achievements of neural networks have far outpaced our theoretical understanding of their properties. Yet, it is hard to imagine that progress can continue indefinitely, without deeper understanding of their fundamental principles and limitations. In this talk I will discuss some recent advances in the mathematics of neural networks and outline what are in my opinion are promising directions for the future research.
How much can one learn a PDE from a single solution data?
When: Mon, March 20, 2023 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Hongkai Zhao (Duke University) - https://scholars.duke.edu/person/Hongkai.Zhao
Abstract: In this presentation, we discuss a few basic questions for PDE learning from observed solution data. Using various types of PDEs as examples, we show 1) how large the data space spanned by all snapshots along a solution trajectory is, 2) if one can construct an arbitrary solution by superposition of snapshots of a single solution, and 3) identifiability of a differential operator from a single solution data on local patches.
Coupling physics-deep learning inversion
When: Mon, March 27, 2023 - 10:30amWhere: Phy 1204 and Online Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Lu Zhang (Columbia University) - https://luzhanghpp.github.io/
Abstract: In recent years, there is great interest in using deep learning to geophysical/medical data inversion. However, direct application of end-to-end data-driven approaches to inversion have quickly shown limitations in the practical implementation. Due to the lack of prior knowledge on the objects of interest, the trained deep learning neural networks very often have limited generalization. In this talk, we introduce a new methodology of coupling model-based inverse algorithms with deep learning for two typical types of inversion problems. In the first part, we present an offline-online computational strategy for coupling classical least-squares based computational inversion with modern deep learning based approaches for full waveform inversion (FWI) to achieve advantages that can not be achieved with only one of the components. An offline learning strategy is used to construct a robust approximation to the inverse operator and utilize it to design a new objective function for the online inversion with new datasets. In the second part, we present an integrated machine learning and model-based iterative reconstruction framework for joint inversion problems where additional data on the unknown coefficients are supplemented for better reconstructions. The proposed method couples the supplementary data with the partial differential equation (PDE) model to make the data-driven modeling process consistent with the model-based reconstruction procedure. The impact of learning uncertainty on the joint inversion results are also investigated.
Data-driven discovery of interacting particle systems via Gaussian process
When: Mon, April 3, 2023 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Sui Tang (University of California, Santa Barbara) - https://web.math.ucsb.edu/~suitang/
Abstract: Interacting particle systems are ubiquitous in science and engineering, exhibiting a wide range of collective behaviors such as flocking of birds, milling of fish, and self-propelled particles. Differential equations are commonly used to model these systems, providing insights into how individual behavior generates collective behaviors. However, quantitatively matching these models with observational data remains a challenge despite recent theoretical and numerical advancements. In this talk, we present a data-driven approach for discovering interacting particle models with latent interactions. Our approach uses Gaussian processes to model latent interactions, providing an uncertainty-aware approach to modeling interacting particle systems. We demonstrate the effectiveness of our approach through numerical experiments on prototype systems and real data. Moreover, we develop an operator-theoretic framework to provide theoretical guarantees for the proposed approach. We analyze recoverability conditions and establish the statistical optimality of our approach. This talk is based on joint works with Jinchao Feng, Charles Kulick, Fei Lu, Mauro Maggioni, and Yunxiang Ren.
What affects the generalization of Neural networks?
When: Mon, April 10, 2023 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Rongrong Wang (Michigan State University) - https://users.math.msu.edu/users/wangron6/
Abstract: Deep neural networks (NN) have led to huge empirical successes in recent years across a wide variety of tasks。 Most deep learning problems are in essence solving an over-parameterized, large-scale non-convex optimization problem. A mysterious phenomenon about NN that attracted much attention in the past few years is why NN generalizes so well.
In this talk, we will begin by reviewing existing theories that attempt to explain this generalization phenomenon when neural networks are trained using the Stochastic Gradient Descent (SGD) algorithm. Building on these results, we will present a new analysis that focuses on the widely-used heavy-ball momentum accelerated SGD (SGD+M) algorithm. Specifically, we will derive the formula for the implicit gradient regularization (IGR) of the SGD+M algorithm and explain its relationship to generalization. We will then use this framework to shed light on previously observed but empirically unexplained behavior of the momentum-accelerated SGD algorithm.This is joint work with Avrajit Ghosh, He Lyu, and Xitong Zhang.
Quantum numerical analysis
When: Mon, April 17, 2023 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Lexing Ying (Stanford University) - https://web.stanford.edu/~lexing/
Abstract: Recent developments in quantum computers have inspired rapid progress in developing quantum algorithms for scientific computing, including examples in numerical linear algebra, partial differential equations, and machine learning. However, the noise of quantum devices and quantum measurements pose new questions in the area of numerical analysis of quantum algorithms. In this talk, I will discuss two of my recent works in this direction: (1) new low-depth algorithms for quantum phase estimation for early fault-tolerant quantum devices and (2) a new robust algorithm for computing phase factors in forming general functions of quantum operators.
On the 2nd AI Wave: Toward Interpretable, Reliable, and Sustainable AI
When: Mon, April 24, 2023 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Jay Kuo (University of Southern California) - https://viterbi.usc.edu/directory/faculty/Kuo/Chung-Chieh
Abstract: Rapid advances in artificial intelligence (AI) in the last decade have been primarily attributed to the wide applications of deep learning (DL) technologies. I view these advances as the first AI wave. There are concerns with the first AI wave. DL solutions are a black box (i.e., not interpretable) and vulnerable to adversarial attacks (i.e., unreliable). Besides, the high carbon footprint yielded by large DL networks is a threat to our environment (i.e., not sustainable). Many researchers are looking for an alternative solution that is interpretable, reliable, and sustainable. This is expected to be the second AI wave. To this end, I have conducted research on green learning (GL) since 2015. GL was inspired by DL. Low carbon footprints, small model sizes, low computational complexity, and mathematical transparency characterize GL. It offers energy-effective solutions in cloud centers and mobile/edge devices. It has three main modules: 1) unsupervised representation learning, 2) supervised feature learning, and 3) decision learning. GL has been successfully applied to a few applications. My talk will present the fundamental ideas of the GL solution and highlight a couple of demonstrated examples.
Machine Learning Developments and Applications in Solid-Earth Geosciences - A Review
When: Mon, May 1, 2023 - 10:30amWhere: Zoom Meeting ID： 927 8056 1489 Password：0900
Speaker: Yunyue Li (Purdue University) - https://www.eaps.purdue.edu/people/profile/li4017.html
Abstract: After decades of low but continuing activity, applications of machine learning (ML) in solid Earth geoscience have exploded in popularity. Based on a special collection on “Machine Learning for Solid Earth Observation, Modeling, and Understanding”, primarily organized by Journal of Geophyscial Research - Solid Earth, I will provide a snapshot of applications ranging from data processing to inversion and interpretation, for which ML appears particularly well suited. Inevitably, there are variations in the degree to which these methods have been developed. I will highlight the successes and discuss the remaining challenges, with examples from my own research group. I hope that the progress seen in some areas will inspire efforts in others. The formidable task of how geoscience can keep pace with developments in ML while ensuring the scientific rigor that our field depends on requires improvements in sensor technology, accelerating rates of data accumulation, and close collaboration among geoscientists, mathematicians, and machine learning engineers. The methods of ML seem poised to play an important role in geosciences for the foreseeable future.