Inhalt und Ziel des Forschungsprojekts:
Unser übergeordnetes Ziel ist es, zum theoretischen Verständnis des maschinellen Lernens beizutragen.
Unser Methodik stammt aus der Informationstheorie. Wie viele Theorien basieren unsere Studien auf einer Vielzahl von statistischen Modellen. Für diese Modelle errechnen wir Erfolgsgarantien für verschiedene Lernstrategien. So entwickeln wir beispielsweise Schranken auf den sogenannten Generalisierungsfehler, d.h., die Differenz zwischen der Erfolgsrate auf den Trainingsdaten und jener auf neuen Testdaten. Diese Schranken geben auch Hinweise, wie man bestehende Algorithmen verbessern könnte.
Wissenschaftlicher und gesellschaftlicher Kontext des Forschungsprojekts:
Unsere Arbeit wird neue Modelle und Algorithmen untersuchen. Diese haben das Potenzial, zur Lösung wichtiger technologischer Probleme beizutragen. Dazu gehören fundamentale Garantien zur Erfolgsquote von Algorithmen des maschinellen Lernens.
There is a longstanding and deep connection between information measures and learning algorithms. This involves superficial aspects as well as rigorous mathematical connections. Probably the most visible connection concerns the use of information measures as imposed figures of merit in learning algorithms. Cross-entropy, for example, is often used as a loss function when optimizing classification models like logistic regression and artificial neural networks. More fundamentally, rigorous connections between information theory and learning algorithms have been established, for example, for principal components analysis (PCA) or multi-armed bandits. To goal of the work proposed here is to further develop such rigorous connections by leveraging recent progress on information measures. Specifically, our interest is, (i), in information measures as an analysis tool for learning and inference algorithms, and (ii), in information measures as an inspiration and guidance in the design of learning and inference algorithms. It is this deep connection and mutual inspiration that is at the basis of the three concrete research vignettes that we propose to investigate:Vignette I - The Generalization Error of Learning Algorithms.Overfitting and the generalization behavior of learning algorithms are important questions. Information measures can shed some light on them, as evidenced in a substantial line of recent work in the community. In recent and on-going work, we have established a set of fundamental bounds. We propose to substantially strengthen this by incorporating more relevant classes of learning algorithms and encompassing more realistic models, as well as leveraging emerging progress on information measures.Vignette II - ``Common Information Components Analysis.''Information measures have inspired many algorithms over time. Leveraging one of our very recent results, we are proposing a new algorithm to accomplish universal unsupervised feature extraction. This algorithm is inspired by an information measure referred to as Wyner's common information, a principled way of identifying the common aspects of multiple random variables. The resulting algorithm extracts features that best express the commonalities between two (or multiple) data sets.Vignette III - The Price of Distributed Learning and Inference.Information measures are powerful tools to quantify communication constraints. If learning and inference have to happen in a distributed setting, split into a number of terminals, this can often be abstracted into a limitation on how much communication is allowed between these terminals. For one and the same inference task, executed on a single machine versus executed on a distributed set of machines, what is the fundamental penalty in quality of inference?While these three vignettes are all different in their specifics, their common thread is the leveraging of recent progress on information measures.
Last updated:18.06.2022
