Research

Neural networks with optimized single-neuron adaptation uncover biologically plausible regularization

V. Geadah, S. Horoi, G. Kerg, G. Wolf, G. Lajoie
PLOS Computational Biology, 2024

Preprint: Goal-driven optimization of single-neuron properties in artificial networks reveals regularization role of neural diversity and adaptation.

paperbioRxiv

Continuous-Time Meta-Learning with Forward Mode Differentiation

T. Deleu, D. Kanaa, L. Feng, G. Kerg, Y. Bengio, G. Lajoie, P.-L. Bacon
Tenth International Conference on Learning Representations (ICLR), 2022

OpenReview

Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization

S. Jastrzebski, D. Arpit, O. Astrand, G. Kerg, H. Wang, C. Xiong, R. Socher, K. Cho, K. Geras
Thirty-eighth International Conference on Machine Learning (ICML), 2021

arXiv

Untangling trade-offs between recurrence and self-attention in artificial neural networks

G. Kerg*, B. Kanuparthi*, A. Goyal, K. Goyette, Y. Bengio, G. Lajoie
Advances in Neural Information Processing Systems (NeurIPS), 2020

arXivcode

Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics

G. Kerg*, K. Goyette*, M.P. Touzel, G. Gidel, E. Vorontsov, Y. Bengio, G. Lajoie
Advances in Neural Information Processing Systems (NeurIPS), 2019

arXivcode

h-detach: Modifying the LSTM gradient towards better optimization

B. Kanuparthi*, D. Arpit*, G. Kerg, R. Ke, I. Mitliagkas, Y. Bengio
Seventh International Conference on Learning Representations (ICLR), 2019

OpenReviewarXivcode

On Neural Architecture Inductive Biases for Relational Tasks

G. Kerg, S. Mittal, D. Rolnick, Y. Bengio, B. Richards, G. Lajoie
arXiv, 2022

arXivcode

Advantages of biologically-inspired adaptive neural activation in RNNs during learning

V. Geadah, G. Kerg, S. Horoi, G. Wolf, G. Lajoie
arXiv, 2020

arXiv

Inductive Biases for Relational Tasks

G. Kerg, S. Mittal, D. Rolnick, Y. Bengio, B. Richards, G. Lajoie
ICLR 2022 Workshop on Objects, Structure and Causality

OpenReviewcode

Top-down optimization recovers biological coding principles of single-neuron adaptation in RNNs

V. Geadah, G. Kerg, S. Horoi, G. Wolf, G. Lajoie
Computational and Systems Neuroscience (COSYNE), 2022

Network-level computational advantages of single-neuron adaptation

V. Geadah, G. Lajoie, G. Kerg, S. Horoi, G. Wolf
Computational and Systems Neuroscience (COSYNE), 2021

Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization

S. Jastrzebski, D. Arpit, O. Astrand, G. Kerg, H. Wang, C. Xiong, R. Socher, K. Cho, K. Geras
NeurIPS 2020 Workshop on Optimization in Machine Learning (OPT)

paper

Guarantees for stable signal and gradient propagation in self-attentive recurrent networks

G. Kerg, B. Kanuparthi, A. Goyal, K. Goyette, Y. Bengio, G. Lajoie
DeepMath 2020 (Conference on the Mathematical Theory of Deep Neural Networks)

Learning Long-term Dependencies Using Cognitive Inductive Biases in Self-attention RNNs

G. Kerg*, B. Kanuparthi*, A. Goyal, K. Goyette, Y. Bengio, G. Lajoie
ICML 2020 Workshop on Inductive Biases, Invariances and Generalization in RL; Montreal AI Symposium (MAIS), 2020

papercode

Safe Screening for Support Vector Machines

J. Zimmert, C. Schröder de Witt, G. Kerg, M. Kloft
NIPS 2015 Workshop on Optimization in Machine Learning (OPT)

paper

Inductive Biases for Efficient Information Transfer in Artificial Networks

Ph.D. thesis, Université de Montréal / Mila, 2023

On Neretin’s group of tree spheromorphisms

Master thesis, M.Sc. in Pure Mathematics, Université Libre de Bruxelles, 2013

thesis

Expansion in groups

Essay for Part III of the Mathematical Tripos, University of Cambridge, 2012

essay

Approaching deep learning from mathematical foundations

Generally Intelligent podcast