# IDeAS

## IDeAS Seminar: Towards de-mystification of deep learning: function space analysis of the representation layers

Speaker:
Shai Dekel, Tel Aviv University
Date:
Sep 19 2017 - 2:30pm
Event type:
IDeAS
Room:
McDonnell Hall, Room 102A
Abstract:

We propose a function space approach to Representation Learning [1] and the analysis of the representation layers in deep learning architectures. We show how to compute a weak-type'  Besov smoothness index that quantifies the geometry of the clustering in the feature space. This approach was already applied successfully to improve the performance of machine learning algorithms such as the Random Forest [2] and tree-based Gradient Boosting [3]. Our experiments demonstrate that in well-known and well-performing trained  networks, the Besov smoothness of the training set, measured in the corresponding hidden layer feature map representation, increases from layer to layer which relates to the unfolding' of the clustering in the feature space. We also contribute to the understanding of generalization [4] by showing how the Besov smoothness of the representations, decreases as we add more mis-labeling to the training data. We hope this approach will contribute to the de-mystification of some aspects of deep learning.

References:

[1] Y. Bengio , A. Courville and P. Vincenty, Representation Learning: A Review and New Perspectives, IEEE Transactions on Pattern Analysis and Machine Intelligence 8 (2013), 1798-1828.

[2] O. Elisha and S. Dekel , Wavelet decompositions of Random Forests - smoothness analysis,sparse approximation and applications, Journal of machine learning research 17 (2016), 1-38.

[3] S. Dekel, O. Elisha and O. Morgan, Wavelet decomposition of Gradient Boosting, preprint.

[4] C. Zhang, S. Bengio, M. Hardt, B. Recht and O. Vinyals, Understanding deep learning requires rethinking generalization, In ICLR 2017 conference proceedings.

Shai serves as Head of AI at WIX and is a visiting associate professor at the school of mathematical sciences at Tel-Aviv University.  For further information see: https://www.shaidekel.com/

## IDeAS Seminar: Stability of some super-resolution problems

Speaker:
Dmitry Batenkov, MIT
Date:
Sep 27 2017 - 2:30pm
Event type:
IDeAS
Room:
224 Fine Hall
Abstract:

The problem of computational super-resolution asks to recover an object from its noisy and limited spectrum. In this talk, we consider two inverse problems of this flavor, mainly from the point of view of stability estimates. In the first problem, we assume that the object's spectrum is a finite sum of exponentials modulated by polynomials (extending the well-researched case where the polynomials are constants). We derive upper bounds on the problem condition number and show that the attainable resolution exhibits Hölder-type continuity with respect to the noise level [1,3]. As an application we consider the approximation of a piecewise-smooth function from its Fourier coefficients. We can show that the asymptotic accuracy of our approach is only dictated by the smoothness of the function between the jumps, even if the jump locations are not known [2].

The second problem is concerned with on-going work on the weighted extrapolation problem on the real line for functions of finite exponential type where we abandon the sparsity assumption. It turns out that the extrapolation range scales logarithmically with the noise level, while the pointwise extrapolation error exhibits again a Hölder-type continuity.

References:

[1] A. Akinshin, D. Batenkov, and Y. Yomdin, “Accuracy of spike-train Fourier reconstruction for colliding nodes,” in 2015 International Conference on Sampling Theory and Applications (SampTA), 2015, pp. 617–621.

[2] D. Batenkov, “Complete algebraic reconstruction of piecewise-smooth functions from Fourier data,” Math. Comp., vol. 84, no. 295, pp. 2329–2350, 2015

[3] D. Batenkov, “Stability and super-resolution of generalized spike recovery,” Applied and Computational Harmonic Analysis, http://dx.doi.org/10.1016/j.acha.2016.09.004.

Dmitry Batenkov is a postdoctoral researcher at the Massachusetts Institute of Technology. His research interests include applied harmonic analysis, approximation theory, numerical analysis, sparse representations, sampling theory and inverse problems.  http://dimabatenkov.info

## IDeAS Seminar

Speaker:
Veit Elser, Cornell University
Date:
Oct 11 2017 - 2:30pm
Event type:
IDeAS
Room:
224 Fine Hall
Abstract:

TBA

## IDeAS Seminar: Iron Age Hebrew Epigraphy in the Silicon Age - An Algorithmic Approach To Study Paleo-Hebrew Inscriptions

Speaker:
Barak Sober, Tel Aviv University
Date:
Oct 18 2017 - 2:30pm
Event type:
IDeAS
Room:
224 Fine Hall
Abstract:

Handwriting comparison and identification, e.g. in the setting of forensics, has been widely addressed over the years. However, even in the case of modern documents, the proposed computerized solutions are quite unsatisfactory. For historical documents, such problems are worsened, due to the inscriptions’ preservation conditions. In the following lecture, we will present an attempt at addressing such a problem in the setting of First Temple Period inscriptions, stemming from the isolated military outpost of Arad (ca. 600 BCE). The solution we propose comprises: (A) Acquiring better imagery of the inscriptions using multispectral techniques; (B) Restoring characters via approximation of their composing strokes, represented as a spline-based structure, and estimated by an optimization procedure; (C) Feature extraction and distance calculation - creation of feature vectors describing various aspects of a specific character based upon its deviation from all other characters; (D) Conducting an experiment to test a null hypothesis that two given inscriptions were written by the same author. Applying this approach to the Arad corpus of inscriptions resulted in: (i) The discovery of a brand new inscription on the back side of a well known inscription (half a century after being unearthed); (ii) Empirical evidence regarding the literacy rates in the Kingdom of Judah on the eve of its destruction by Nebuchadnezzar the Babylonian king.

Barak Sober received his B.Sc. degree in Mathematics and Philosophy from Tel Aviv University, Israel, in 2006. He received his M.Sc. in Applied Mathematics (summa cum laude) on the topic of “Handwritten Character Stroke Reconstruction” in 2013. He is currently pursuing his PhD in Applied Mathematics on the topic of “Approximation of Manifolds from Scattered Data”. His interests include approximation theory, machine learning, image processing and archaeology.

## IDeAS Seminar

Speaker:
Nicolas Garcia Trillos, Brown University
Date:
Oct 25 2017 - 2:30pm
Event type:
IDeAS
Room:
224 Fine Hall
Abstract:

TBA

## IDeAS Seminar

Speaker:
Nadav Dym, Weizmann Institute
Date:
Nov 8 2017 - 2:30pm
Event type:
IDeAS
Room:
224 Fine Hall
Abstract:

TBA

## IDeAS Seminar

Speaker:
Joe Kileel, Princeton University
Date:
Nov 15 2017 - 2:30pm
Event type:
IDeAS
Room:
224 Fine Hall
Abstract:

TBA

## IDeAS Seminar

Speaker:
Gal Mishne, Yale University
Date:
Nov 29 2017 - 2:30pm
Event type:
IDeAS
Room:
224 Fine Hall
Abstract:

TBA

## IDeAS Seminar

Speaker:
Ti-Yen Lan, Cornell University
Date:
Dec 6 2017 - 2:30pm
Event type:
IDeAS
Room:
224 Fine Hall
Abstract:

TBA