This course will present the theoretical basis of statistical learning, with a specific focus on binary classification (non-binary classification and regression are discussed throughout the lectures, but non-supervised learning is not covered). The students will learn the fundamental and relevant concepts of class complexity, the associated risk bounds and the limits of Empirical Risk Mininimization. Then, they will discover the theory background behind the classical families of algorithms: Support Vector Machines, Boosting and Neural Nets.
Those algorithms are going to be further studied and/or implemented with 3 practical sessions (essentially on Python)
- Main problem, examples and risk concepts
- Empirical Risk Minimization (finite class, VC-dimension, convexification, chaining)
- Support Vector Machines (RKHS and Kernel tricks)
- Neural Nets (Expressivity, Complexity and Optimization)
Y. Mansour, Machine Learning: Foundations, Tel-Aviv University, 2013
P. Rigollet, Mathematics of Machine Learning, MIT, 2015
A. Ng, Machine Learning, Stanford, 2015
S. Kakade and A. Tewari, Topics in Artificial Intelligence, TTIC, 2008
L. Devroye, L. Gyorfi and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Springer 1996.
T. Hastie, R. Tibshirani and J. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer 2009.