STA 4273 Winter 2025: Modern Learning Theory
This course covers several topics in modern machine learning theory.
Topics may include: Uniform Convergence, Complexity measures, Double descent risk curves, Neural Tangent Kernel, Feature Learning in Neural Networks, Neural Tangent Kernel, Neural Scaling Laws, Edge of Stability, Transformers, Mean Field Langevin Dynamics, Diffusion Models, Edge of Stability, etc. More details can be found in syllabus. Please also sign up for Piazza.
This class requires a good informal knowledge of probability theory, linear algebra, real analysis (at least Masters level). Homework 0 is a good way to check your background.
Students will be expected to present a paper, prepare code notebooks, and complete a final project on a topic of their choice.
Announcements:
- Assignment is out and due on 2/120 23:59, to be submitted through crowdmark.
- First lecture is on Jan 9th!
Instructors: Murat A. Erdogdu
- Email: sta4273prof@cs.toronto.edu
- Office hours: Th 18:00-19:00 at BA 5256
Teaching Assistants: TBD
- Email: sta4273ta@cs.toronto.edu
Time & Location:
| Section | Room | Lecture time |
|---|---|---|
| L0101 | BA 1170 | Th 16-18 |
Lectures and timeline
| Week | Topics | Lectures | Reading | Timeline |
|---|---|---|---|---|
| 1 | Introduction | lecture 1 | syllabus | |
| 2 | Uniform convergence | lecture 2 | Sec 2 of BMR25 | |
| 3 | Double-descent Risk Curves | lecture 3 | Bac24 & HMRT19 | Assignment out |
| 4 | Neural Tangent Kernel | lecture 4 | Topic due | |
| 5 | Feature Learning in Neural Networks | lecture 5 | Assignment due | |
| 6 | Neural Scaling Laws | lecture 6 | ||
| 7 | Reading week | |||
| 8 | Linear Transformers | lecture 7 | Proposals due | |
| 9 | Log-concave sampling | lecture 8 | ||
| 10 | Mean Field Langevin Dynamics | lecture 9 | ||
| 11 | Diffusion Models | lecture 10 | ||
| 12 | Edge of Stability | lecture 11 | ||
| 13 | - | lecture 12 | Project due |
Assignments and Grading
- 15% – Assignment (due 20 Feb): Students solve a problem set.
- 30% – Paper presentation and report (due at the start of presentation): Students pick a topic (due 13 Feb), instructor picks a paper on that topic.
- 5% – Project proposal (due 13 Feb): Students pick a topic different than their presentation topic.
- 50% – Project report (due 9 Apr)
More details will be released later.
Computing Resources
For the assignments, we will use Python, and libraries such as NumPy, SciPy, and scikit-learn. You have two options:
- The easiest option is run everything on colab.
- Alternatively, you can install everything yourself on your own machine.
- If you don’t already have python, install using Anaconda.
- Use pip to install the required packages
pip install scipy numpy autograd matplotlib jupyter sklearn
- For those unfamiliar with Numpy, there are many good resources, e.g. Numpy tutorial and Numpy Quickstart.