Surprisingly, many optimisation phenomena observed in complex neural networks also appear in so-called 2-layer diagonal linear networks. This rudimentary architecture—a two-layer feedforward linear network with a diagonal inner weight matrix—has the advantage of revealing key training characteristics while keeping the theoretical analysis clean and insightful.In this talk, I’ll provide an overview of various theoretical results for this architecture, while drawing connections to experimental observations from practical neural networks. Specifically, we’ll examine how hyperparameters such as the initialisation scale, step size, and batch size impact the optimisation trajectory and influence the generalisation performance of the recovered solution.

On étudie un processus d’apprentissage par renforcement, pour la recherche de plus courts chemins dans un graphe, dans lequel des fourmis partent d’un nid (aléatoire, N1 ou N2) et font une marche aléatoire (pondérée par les poids des arêtes) jusqu’à une source de nourriture F. À leur retour, elles renforcent les arêtes (en ajoutant 1 à leur poids) appartenant au chemin aller auquel on a enlevé les boucles inutiles. Ce modèle a déjà été étudié sur divers graphes dans le cas où le nid est déterministe, notamment les graphes séries-parallèles, mais aussi pour d’autres politiques de renforcements (articles de Kious, Mailler et Schapira). Nous étudions le cas à deux nids, dans des graphes obtenus en joignant trois graphes séries-parallèles pour former un triangle.

On montre que les poids des arêtes (normalisés) convergent, vers des variables aléatoires nulles si les arêtes associées n’appartiennent pas à un plus court chemin d’un sommet de {N1 , N2 , F } à un autre.

Nous présenterons plusieurs outils utiles pour prouver cette convergence, notamment la comparaison avec des processus d'urnes, et quelques résultats sur les approximations stochastiques.

La présentation se basera sur un travail en commun avec Cécile Mailler.

Recurrent neural networks with low-rank connectivity matrices are general and tractable models of collective dynamics in large networks. This class of models dates back to the seminal works of J. Hopfield (1982) and S. Amari (1972), and it still plays an instrumental role in computational neuroscience today. To highlight the analytical tractability of these models, I will first review some recent theoretical results concerning the case where the low-rank connectivity is random, the rank is kept fixed, and the number of neurons tends to infinity. In this case, we find that (i) the dynamics of the network converges to a neural field equation, (ii) the dynamics can be reduced to a latent, low-dimensional dynamical system, and (iii) the latent dynamics can be fully solved in certain special cases. In the second part of the presentation, I will show that low-rank connectivity is associated with remarkable concentration of measure phenomena in networks of biological neurons. Considering a feedforward network setup where neurons in the first layer, modelled as Cox processes, transmit stochastic spikes, I will present a theorem stating that the network can behave as if each spiking neuron were transmitting its subthreshold membrane potential as both the rank of the connectivity and the number of neurons tend to infinity. This result could explain how, at the network level, neurons can transmit their subthreshold membrane potential fluctuations through sparse spikes. The proof of the theorem involves the so-called thin shell phenomenon, a well-known concentration phenomenon in high-dimensional probability.

In recent years, considerable progress has been made in implementing decision support procedures based on machine learning methods through the use of very large databases and learning algorithms. In many application areas, the available databases are modest in size, raising the question of whether it is reasonable, in this context, to seek to develop powerful tools based on machine learning techniques. This presentation introduces models that leverage various types of knowledge through pre-trained alternative models, targeted observations, or physics in order to implement effective machine learning models in a context of data scarcity. Several industrial applications are used to illustrate benefits of these approaches.

https://www.phnmsbern.ch/public/ph/bilder/Portraits/Mitarbeitende/250801_kirchner_cv.pdf