This course covers several topics in classical machine learning theory.

Topics may include: Asymptotic statistics, Uniform Convergence, Generalization, Complexity measures, Kernel Methods, Online Learning, Sampling. More details can be found in syllabus. Please also sign up for Piazza.

This class requires a good informal knowledge of probability theory, linear algebra, real analysis (at least Masters level). Homework 0 is a good way to check your background.

- Zoom links for the lecture and office hours will be sent out through quercus every week.

- Email: csc2532prof@cs.toronto.edu
- Office hours: F 18:00-19:00 online

- Email: csc2532tas@cs.toronto.edu

Section | Room | Lecture time |
---|---|---|

L0101 | online | F 16-18 |

No required textbooks. Suggested reading will be posted after each lecture (See lectures below).

- (ITIL) MacKay (2003) Information Theory, Inference, and Learning Algorithms
- (UML) Shalev-Shwartz, Ben-David (2014) Understanding Machine Learning: From Theory to Algorithms
- (HDP) Vershynin (2018) High Dimentional Probability
- (HDS) Wainwright (2019) High Dimentional Statistics

Week | Day | Topics and Lecture notes | Lectures | Recordings | Timeline |
---|---|---|---|---|---|

1 | 1/14 | Introduction & Warm-up: Gaussian Mean Estimation | notes 1 | lecture 1 | syllabus |

2 | 1/21 | Exponential Families and Information Inequality | |||

3 | 1/28 | Asymptotic statistics | hw1 out | ||

4 | 2/04 | Uniform convergence & Generalization | |||

5 | 2/11 | Covering with epsilon-nets | hw1 due & hw2 out | ||

6 | 2/18 | Rademacher complexity I | |||

7 | 2/25 | Rademacher complexity II | hw2 due | ||

8 | 3/04 | Combinatorial Measures of Complexity | hw3 out | ||

9 | 3/11 | Chaining and Dudley’s theorem | project proposal due | ||

10 | 3/18 | Algorithmic Stability | hw 3 due | ||

11 | 3/25 | Midterm | |||

12 | 4/01 | Kernel Methods I | |||

13 | 4/08 | Kernel Methods II | Final reports due |

Homework # | Out | Due | TA Office Hours |
---|---|---|---|

Homework 0 - V0 | 1/11 | - | - |

Homework 1 - V0 | 1/28 | 2/11 | - |

Homework 2 - V0 | 2/11 | 2/25 | - |

Homework 3 - V0 | 3/01 | 3/18 | - |

Latex template can be found here.

Your project goal is to read and write a comprehensive review of a theoretical machine learning paper, and understand the main building blocks.

**Project Inspiration:**
You can go through recent papers on COLT, NeurIPS, ICML, ICLR, JMLR to get project ideas and pick a paper to review.

List of suggested papers will be posted here.

Latex template for reports can be found here.

For the homework assignments, we will use Python, and libraries such as NumPy, SciPy, and scikit-learn.

- If you don’t already have python, install it. We recommend using Anaconda. You can also install python directly if you know how.
- Use pip to install the required packages
`pip install scipy numpy matplotlib jupyter sklearn`