CSE 515T: Bayesian Methods in Machine Learning – Fall 2024
Instructor: Professor Roman Garnett
TA: Yehu Chen
Time/Location: Monday/Wednesday 10–11:20am, Seigle 301
Office hours (Garnett): by appointment, always on Slack
syllabus
Slack invite link
Description
This course will cover modern machine learning techniques from a Bayesian probabilistic perspective. Bayesian probability allows us to model and reason about all types of uncertainty. The result is a powerful, consistent framework for approaching many problems that arise in machine learning, including parameter estimation, model comparison, and decision making. We will begin with a high-level introduction to Bayesian inference, then proceed to cover more-advanced topics.
Project
You can find detailed information about the project here.
Lectures
Lecture 1: Introduction to the Bayesian Method
Wednesday, 28 August 2024
lecture notes
Additional Resources:
- Book: Bishop PRML: Section 1.2 (Probability theory)
- Book: Barber BRML: Chapter 1 (Probabilistic reasoning)
- Book: Garnett BayesOpt: Section 1.2 (The Bayesian Approach)
- Book: Bayesian Method for Hackers (Cam Davidson Pilon) Great high-level overview from an atypical perspective!
- Video: Introduction to Machine Learning (Nando de Freitas)
- Video: Bayesian Inference I (Zoubin Ghahramani) (the first 30 minutes or so)
- Video: Machine Learning Coursera course (Andrew Ng) The first week gives a good general overview of machine learning and the third week provides a linear-algebra refresher.
Lecture 2: Bayesian Inference I (coin flipping)
Wednesday, 4 September 2024
lecture notes
Additional Resources:
Lecture 3: The Gaussian Distribution
Wednesday, 11 September 2024
lecture notes – see Garnett BayesOpt: Appendix A
Additional Resources:
Lecture 4: Bayesian Inference II (hypothesis testing and summarizing distributions)
Monday, 16 September 2024
lecture notes
Additional Resources:
- Article: "The Fallacy of Placing Confidence in Confidence Intervals" available here or here
Lecture 5: Bayesian Inference III (decision theory)
Wednesday, 18 September 2024
lecture notes
Additional Resources:
- Book: Bishop PRML: Section 1.5 (Decision theory)
- Book: Berger Chapter 1 (Basic concepts), Section 4.4 (Bayesian decision theory)
- Book: Robert Section 4.2 (Bayesian decision theory)
- Videos: YouTube user mathematicalmonk has a great series of machine-learning lectures available. Chapter 11 concerns decision theroy.
Lecture 6: Bayesian Linear Regression
Monday, 22 September 2024
lecture notes
Additional Resources:
- Book: Bishop PRML: Section 3.3 (Bayesian Linear Regression).
- Book: Barber BRML: Section 18.1 (Regression with Additive Gaussian Noise).
- Book: Rasmussen and Williams GPML: Section 2.1 (Weight-space View), available here.
- Video: YouTube user mathematicalmonk has an entire section devoted to Bayesian linear regression. See ML 10.1–7 here.
- Videos: Nando de Freitas has a series of lectures on Bayesian linear regression. Part one is here, and part two is here.
Lecture 7: Bayesian Model Selection
Wednesday, 25 September 2024
lecture notes
Additional Resources:
- Book: Bishop PRML: Section 3.4 (Bayesian Model Comparison).
- Book: Barber BRML: Chapter 12 (Bayesian Model Selection).
- Book: MacKay ITILA: Chapter 28 (Occam's Razor and Model Comparison).
- Book: Garnett BayesOpt: Chapter 4 (Model Assessment, Selection, and Averaging).
- Video: YouTube user mathematicalmonk has a lecture about Bayesian model selection (some nearby videos are related as well).
Lecture 8: The Kernel Trick
Monday, 30 September 2024
lecture notes
Additional Resources:
- Book: Rasmussen and Williams GPML: Chapter 2 through 2.1 (Weight-space View), available here.
Lecture 9: Bayesian Logistic Regression / The Laplace Approximation
Wednesday, 2 October 2024
lecture notes
Additional Resources:
- Book: Bishop PRML: Chapter 4 (Linear Models for Classificaiton).
- Book: Barber BRML: Section 18.2 (Classification).
- Book: Rasmussen and Williams GPML: Sections 3.1 and 3.2 (Classification Problems and Linear Models for Classification), available here.
- Video: YouTube user mathematicalmonk has a lecture about Bayesian logistic regression.
Lecture 10: Gaussian Process Regression
Wednesday, 9 October 2024
lecture notes &ndsh; see Garnett BayesOpt: Chapter 2
lecture slides
Additional Resources/Notes:
- Book: Rasmussen and Williams GPML: Sections 2.2 – 2.5, available here.
- Book: Barber BRML: Chapter 19 (Gaussian processes).
- Video: Nando de Freitas has a lecture here.
- Video: Philipp Hennig has a series of lectures from the 2013 Machine Learning Summer School; part one is here. The slides, which have some cool animations, are available here.
- Video: Carl Rasmussen has a two-part introduction to Gaussian processes here.
- Video: David MacKay gave an introduction to Gaussian processes here.
Lecture 11: Kernels
Monday, 14 October 2024
lecture notes – see Garnett BayesOpt: Chapter 3
Resources/Notes:
Lecture 12: A Quick Interlude
Wednesday, 16 October 2024
Additional Resources/Notes:
- Slides: Roman Garnett (+ Simeon Bird, Shirley Ho, and Jeff Schneider), Finding Galaxies in the Shadows of Quasars, available here.
- Paper: Accompanying paper available here.
Lecture 13: Bayesian Optimization
Monday, 21 October 2024
lecture slides
Additional Resources/Notes:
- Book: Garnett, Bayesian Optimization, available here.
- Paper: Niranjan Srinivas, Andreas Krause, Sham Kakade, and Mattias Seeger discuss the GP-UCB algorithm (including theoretical results!) in this landmark paper.
- Paper: Jasper Snoek, Hugo Larochelle, and Ryan P. Adams discuss the AutoML application of Bayesian optimization here.
Lecture 14: Bayesian Quadrature
Wednesday, 23 October 2024
lecture notes
Additional Resources/Notes:
- Slides: David Duvenaud has a set of slides introucing Bayesian quadrature, available here.
- Paper: Persi Diaconis's essay "Bayesian Numerical Analysis" from 1998 provides an excellent introduction.
- Paper: Carl Rasmussen and Zoubin Ghahramani discuss Bayesian quadrature under the name "Bayesian Monte Carlo" in this paper. Many references therein are also interesting, especially the provocatively titled Monte Carlo is fundamentally unsound by Anthony O'Hagan.
- Paper: Tom Minka wrote a report on "Deriving quadrature rules from Gaussian processes," available here.
Lecture 15: Sequential Decision Theory / Active Search
Wednesday, 6 November 2024
lecture slides
Additional Resources/Notes:
- Book: Berger, Sequential Decision Theory and Bayesian Analysis: Section 7.4 (Bayesian Sequential Analysis).
- Book: DeGroot, Optimal Statistical Decisions: Part 4 (Sequential Decisions).
- Book: Garnett, Bayesian Optimization: Section 11.11 (Non-Gaussian Observation Models and Active Search).
- Paper: Garnett, et al., Bayesian Optimal Active Search and Surveying (ICML 2012).
- Paper: Jiang, et al., Efficient Nonmyopic Active Search (ICML 2017).
Lecture 16: Importance Sampling, MCMC
Wednesday, 13 November 2024
lecture slides (from Iain Murray's introduction at the 2009 Machine Learning Summer School)
Additional Resources/Notes:
- Video: You can watch Iain Murray present the slides himself here.
- Book: Barber BRML: Sections 27.4 (Markov Chain Monte Carlo (MCMC)), 27.3 (Gibbs Sampling), and 27.6 (Importance Sampling).
- Book: Bishop PRML: Sections 11.2 (Markov Chain Monte Carlo) and 11.3 (Gibbs Sampling).
- Videos: YouTube user mathematicalmonk has a chapter devoted to sampling methods (#17), beginning here.
Lecture 17: Practical Issues with GPs
Monday, 18 November 2024
lecture notes – see Garnett BayesOpt: Chapter 9
Lecture 18: Bayesian additive regression trees (BART)
Wednesday, 20 November 2024
Additional Resources/Notes:
- Paper: The original paper is here.
Resources
Books
There is no required book for this course. That said, there are a wide variety of machine-learning books available, some of which are available for free online. The following books all have a Bayesian slant to them:
- Pattern Recognition and Machine Learning (PRML) by Christopher M. Bishop. Covers many machine-learning topics thoroughly. Definite Bayesian focus. Can also be very mathematical and take some effort to read.
- Bayesian Reasoning and Machine Learning (BRML) by David Barber. Geared (as much as a machine-learning book can be!) towards computer scientists. Lots of material on graphical models. Freely available online.
- Gaussian Processes for Machine Learning (GPML) by Carl Rasmussen and Christopher Williams. Excellent reference for Gaussian processes. Freely available online.
- Bayesian optimization by me! Although the focus is on Bayesian optimization, there is a lot of background material on Gaussian processes as well. Freely available online.
- Information Theory, Inference, and Learning Algorithms by David J. C. Mackay. Very strong focus on information theory. If you have a background in physics or are interested in information theory, this is the book for you. Freely available online.
For a more-frequentist perspective, check out the excellent The Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani, and Jerome Friedman. Freely available online.
Websites
- I will post the source for lecture notes, demo code, etc. on this GitHub page. Even the source for the syllabus and this website are there.
- I have created a Slack for this class (invite link). Please post any questions about lectures, homework, etc here! Chances are that someone else has the same question and we can all benefit from a public discussion. If you have a question just for me and/or me and the TA, please also send a direct message via Slack rather than emailing us directly.
- There are several relevant courses available on Coursera. Coursera gives you access to video lecture series, often from world experts, all available for free! In particular, the following three courses are all presented by leaders in the field:
Other
The Matrix Cookbook by Kaare B. Petersen and Michael S. Pedersen can be incredibly useful for helping with tricky linear alegbra problems!