RIT on Deep Learning Archives for Fall 2023 to Spring 2024
Machine Unlearning: Linear Filtration for Logit-based Classifiers
When: Fri, September 30, 2022 - 1:00pmWhere: Kirwan Hall 1310
Speaker: Gonzalo Benavides (AMSC) -
Abstract: Recently enacted legislation grants individuals certain rights to decide in what fashion their personal data may be used, and in particular a "right to be forgotten". This poses a challenge to machine learning: how to proceed when an individual retracts permission to use data which has been part of the training process of a model? From this question emerges the field of machine unlearning, which could be broadly described as the investigation of how to "delete training data from models". Our work complements this direction of research for the specific setting of class-wide deletion requests for classification models (e.g. deep neural networks). As a first step, we propose linear filtration as a intuitive, computationally efficient sanitization method. Our experiments demonstrate benefits in an adversarial setting over naive deletion schemes.
On the Universality of Invariant Networks
When: Fri, October 14, 2022 - 1:00pmWhere: Kirwan Hall 1310
Speaker: Shashank Sule (AMSC) -
Abstract: Constraining linear layers in neural networks to respect symmetry transformations from a group G is a common design principle for invariant networks that has found many applications in machine learning. In this paper, we consider a fundamental question that has received very little attention to date: Can these networks approximate any (continuous) invariant function? We tackle the rather general case where G≤Sn (an arbitrary subgroup of the symmetric group) that acts on \Rn by permuting coordinates. This setting includes several recent popular invariant networks. We present two main results: First, G-invariant networks are universal if high-order tensors are allowed. Second, there are groups G for which higher-order tensors are unavoidable for obtaining universality. G-invariant networks consisting of only first-order tensors are of special interest due to their practical value. We conclude the paper by proving a necessary condition for the universality of G-invariant networks that incorporate only first-order tensors. Lastly, we propose a conjecture stating that this condition is also sufficient.
ADef: an Iterative Algorithm to Construct Adversarial Deformations
When: Fri, October 21, 2022 - 1:00pmWhere: Kirwan Hall 1310
Speaker: Matthias Wellershoff (MATH) -
Abstract: While deep neural networks have proven to be a powerful tool for many recognition and classification tasks, their stability properties are still not well understood. In the past, image classifiers have been shown to be vulnerable to so-called adversarial attacks, which are created by additively perturbing the correctly classified image. In this paper, we propose the ADef algorithm to construct a different kind of adversarial attack created by iteratively applying small deformations to the image, found through a gradient descent step. We demonstrate our results on MNIST with convolutional neural networks and on ImageNet with Inception-v3 and ResNet-101.
Group-invariant max filtering
When: Fri, October 28, 2022 - 1:00pmWhere: Kirwan Hall 1310
Speaker: Stratos Tsoukanis (MATH) -
Abstract: Given a real inner product space V and a group G of linear isometries, we construct a family of G-invariant real-valued functions on V that we call max filters. In the case where V=Rd and G is finite, a suitable max filter bank separates orbits, and is even bilipschitz in the quotient metric. In the case where V=L2(Rd) and G is the group of translation operators, a max filter exhibits stability to diffeomorphic distortion like that of the scattering transform introduced by Mallat. We establish that max filters are well suited for various classification tasks, both in theory and in practice.
Deep Haar Scattering Networks
When: Fri, November 11, 2022 - 1:00pmWhere: Kirwan Hall 1310
Speaker: Canran Ji (MATH) -
A Universal Approximation Theorem of Deep Neural Networks for Expressing Probability Distributions
When: Fri, November 18, 2022 - 1:00pmWhere: Kirwan 1310
Speaker: Sanghoon Na (AMSC) -
Abstract: This paper studies the universal approximation property of deep neural networks for representing probability distributions. Given a target distribution π and a source distribution p_z both defined on Rd, we prove under some assumptions that there exists a deep neural network g:Rd→R with ReLU activation such that the push-forward measure (∇g)#p_z of p_z under the map ∇g is arbitrarily close to the target measure π. The closeness are measured by three classes of integral probability metrics between probability distributions: 1-Wasserstein distance, maximum mean distance (MMD) and kernelized Stein discrepancy (KSD). We prove upper bounds for the size (width and depth) of the deep neural network in terms of the dimension d and the approximation error ε with respect to the three discrepancies. In particular, the size of neural network can grow exponentially in d when 1-Wasserstein distance is used as the discrepancy, whereas for both MMD and KSD the size of neural network only depends on d at most polynomially. Our proof relies on convergence estimates of empirical measures under aforementioned discrepancies and semi-discrete optimal transport.