Model Selection
Requirements:
- Minimal background in mathematics and statistic
- Analysis and calculus (integral, derivatives, study of functions, … )
- Basic statistics concepts
- Basic expertise with Python and Jupyter Notebook
Course Material all the material is available at this page.
| Lesson | Support Slides | Notebook | Additional Material | References | |
|---|---|---|---|---|---|
| Day 1 | Introduction | Basic Probability Models and Sampling in Python | / | / | |
| Day 2 | / | Data Generation - Regression & Classification | / | [Gu2003], [Gu2007] | |
| Day 3 | / | Bias and Variance | / | [HTF2001]:Ch7, [Bis2006]:Ch1,Ch3, [Gem1992] | |
| Day 4 | / | Boostrap and bagging | / | [Brei1996], [Efr1986], [HTF2001]:Ch7 | |
| Day 5 | / | Cross-validation | / | [HTF2001]:Ch7 | |
| Day 6 | / | Classroom Test | / | / | |
| Day 7 | / | Cross-validation II | / | [HTF2001]:Ch7 | |
| Day 8 | / | Information Criteria I | / | [McE2016]:Ch6 | \ |
| Day 9 | / | Information Criteria II | / | [McE2016]:Ch6 |
References
- [Gu2003] Design of experiments for the NIPS 2003 variable selection benchmark. I. Guyon, 2003. link
- [Gu2007] Competitive baseline methods set new standards for the NIPS 2003 feature selection benchmark. I. Guyon, J. Li, T. Mader, P.A. Pletscher, G. Schneider, M. Uhr. Pattern Recognition Letters, 28, 1438-1444, 2007.
- [HTF2001] The Elements of Statistical Learning. T. Hastie, R. Tibshirani, and J. Friedman. Springer Series in Statistics Springer New York Inc., New York, NY, USA, 2001.
- [Bis2006] Pattern Recognition and Machine Learning. C.M. Bishop. Springer-Verlag Berlin, Heidelberg, DE, 2006.
- [Gem1992] Neural networks and the bias/variance dilemma. S. Geman, E. Bienenstock, R Doursat. Neural Computation, 4:2, 1-58, 1992.
- [Brei1996] Bagging predictors. L. Breiman. Machine learning, 24(2), 123-140, 1996.
- [Efr1986] Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. B. Efron, R. Tibshirani, Statistical science, 54-75. 1986.
- [Koh1995] A study of cross-validation and bootstrap for accuracy estimation and model selection. R. Kohavi. IJCAI, 14:2, 1995.
- [Rao2008] On the dangers of cross-validation. An experimental evaluation. R. Barat Rao, G. Fung, and R. Rosales. Proceedings of the 2008 SIAM International Conference on Data Mining. Society for Industrial and Applied Mathematics, 2008.
- [McE2016] Statistical Rethinking. A Bayesian Course with Examples in R and Stan. R. McElreath. T&F Crc Press, 2016.
- [Wag2004] AIC model selection using Akaike weights.</i> E.J. Wagenmakers , S. Farrell. Psychon Bull Rev. 11(1):192-6, 2004.
- [Sym2011] A brief guide to model selection, multimodel inference and model averaging in behavioural ecology using Akaike’s information criterion. M.R. Symonds, A. Moussalli, A. Behavioral Ecology and Sociobiology, 65(1), 13-21, 2011.