Course Syllabus
This course is co-linked with CSE392(#63165) and M393C(#54664)
Instructor:
Professor Chandrajit Bajaj
- Lecture Hours – Mon, Wed- 2:00 - 3:15 pm. GDC 1.406. If Online, go to Zoom panel.
- Office hours -- Tue 1:00 p.m. - 3:00 p.m. or by appointment ( Zoom Links to an external site. or POB 2.324)
- Contact: bajaj@cs.utexas.edu bajaj@oden.utexas.edu
NOTE: All questions related to class should be posted through Piazza. Here is the link to register for Piazza: Links to an external site. You can also join via the Piazza Tab on the Canvas course page.
Teaching Assistant
Yi Wang
- Office hours – Thur 1:00 p.m. - 3:00 p.m. ( Zoom Links to an external site. or POB 2.102)
- Contact: yiwang@oden.utexas.edu
Note: Please attempt to make reservations a day before to avoid conflicts.
Course Motivation and Synopsis
This Spring and Fall course this year is on the design and performance analysis of optimally controlled, aka reinforcement learned statistical machine learning algorithms, trained, verified and validated on filtered, noisy observation data distributions collected from various multi-scale dynamical systems. The principal performance metrics will be on online and energy efficient training, verification and validation protocols that achieve principled and stable learning for maximal generalizability . The emphasis will be on possibly corrupted data and/or the lack of full information for the learned stochastic decision making dynamic algorithmic process. Special emphasis will also be given to the underlying mathematical and statistical physics principles of Free Energy and stochastic Hamiltonian dyamics . Students shall thus be exposed to the latest stochastic machine learning modeling approaches for optimized decision-making, multi-player games involving stochastic dynamical systems and optimal stochastic control. These latter topics are foundational to the training of multiple neural networks (agents) both cooperatively and in adversarial scenarios to optimize the learning process of all the agents.
An initial listing of lecture topics and reference material are given in the syllabus below. This is subject to some modification, given the background and speed at which we cover ground. Homework exercises shall be given almost bi-weekly. Assignment solutions that are turned in late shall suffer a 10% per day reduction in credit and a 100% reduction once solutions are posted. There will be a mid-term exam in class. The exam content will be similar to the homework exercises. A list of topics will also be assigned as take-home final projects to train the best of scientific machine-learned decision-making (agents). The projects will involve modern ML programming, an oral presentation, and a written report submitted at the end of the semester.
This project shall be graded and be in lieu of a final exam.
The course is open to graduate students in all disciplines. Those in the 5-year master's program students, and in the CS, CSEM, ECE, MATH, STAT, PHYS, CHEM, and BIO, are welcome. You’ll need an undergraduate level background in the intertwined topics of algorithms, data structures, numerical methods, numerical optimization, functional analysis, algebra, geometry, topology, statistics, stochastic processes . You will need programming experience (e.g., Python ), at a CS undergraduate senior level.
Course Reference Material (+ reference papers cited in lectures )
- [B1] Chandrajit Bajaj (frequently updated) A Mathematical Primer for Computational Data Sciences
- [PML1] Kevin Murphy Probabilistic Machine Learning: An Introduction. Download Probabilistic Machine Learning: An Introduction.
- [PML2] Kevin Murphy Probabilistic Machine Learning: Advanced Topics. Download Probabilistic Machine Learning: Advanced Topics.
- [M1] Peter S. Maybeck Stochastic Models, Estimation and Control Volume 1 Download Stochastic Models, Estimation and Control Volume 1
- [M2] Peter S. Maybeck Stochastic Models, Estimation and Control Volume 2 Download Stochastic Models, Estimation and Control Volume 2
- [M3] Peter S. Maybeck Stochastic Models, Estimation and Control Volume 3 Download Stochastic Models, Estimation and Control Volume 3
- [MU] Michael Mitzenmacher, Eli Upfal Probability and Computing (Randomized Algorithms and Probabilistic Analysis) Download Probability and Computing (Randomized Algorithms and Probabilistic Analysis)
- [SB] Richard Sutton, Andrew Barto Reinforcement Learning
- [SD] Shai Shalev-Shwartz, Shai Ben-David Understanding Machine Learning, From Theory to Algorithms
- [Basar] Tamer Basar Lecture Notes on Non-Cooperative Game Theory.
- [BHK] Avrim Blum, John Hopcroft, and Ravindran Kannan. Foundations of Data Science
- [BV] Stephen Boyd and Lieven Vandenberghe Convex Optimization.
- Extra reference materials.
TENTATIVE COURSE OUTLINE (in Flux).
Date | Topic | Reading | Assignments |
Mon 01-13-2025 |
1. Introduction to High-Dimensional Spaces, Belief, and Decision-Making Spaces, [Lec1 Download Lec1] Dynamical Systems and Deep Learning [notes] Download [notes]
|
[PML1] Ch 1.1, 1.2, 1.3, 1.4 |
|
Wed 01-15-2025 |
2. Stochastic Machine Learning: Entropy, Distributional Estimates [notes] Download notes] Geometry of Norms and Approximations [notes Download notes]; |
[PML1] Ch 2.1, 2.2, 2.3, 2.4, 2.5, 2.6 [PML1] Ch 4.1, 4.2, 4.5, 4.7, 6.1, 6.2. |
[A1 Download A1] with [latex template Download latex template] out today; |
Fri 01-17-2025 (GDC 4.302) |
3. Multivariate Gaussians, Infinite Dimensional Spaces and Gaussian Processes [Notes] [Lec3] Download [Lec3] Introduction to Stochastic Processes [Supp Notes] Download [Supp Notes]
|
[M1] Ch 3 [PML1] Ch 17.2 [PML2] Ch 18.1,18.2,18,3, 18.5 |
|
Mon 01-27-2025 |
4. Additional notes for Gaussian Processes [notes Download notes] Bayesian Deep Learning [notes] Univariate Time Series Analysis [supp notes] |
[PML1] Ch 17.2 [PML2] Ch 18.1,18.2,18,3, 18.5
|
|
Wed 01-29-2025 |
5. Time Series Bayesian Regression with Deep Gaussian Processes, Kalman Filtering [Supp notes][Supp Notes2] Basics of Kalman Filters [Lec5 Download Lec5] Illustrated Kalman Filters [notes Download notes]
|
|
[A2 Download A2] will be out on Friday; |
Mon 02-03-2025 |
6. MonteCarlo vs Quasi Monte-Carlo Sampling: Relationship to Integration Error H-K Inequality [Lec7 Download Lec7][notes Download notes]
|
[BHK] Ch 2.1-2.6 [PML2] Ch 11 |
|
Wed 02-05-2025 |
7. Statistical Machine Learning 2: Introduction to Markov Chains, Page Rank, MCMC [Lec7] |
[PML1] Ch 3.6 [PML2] Ch 2.6, 4.2, 7.4.5, 7.4.6, |
|
Mon 02-10-2025 |
8. Sampling in High-Dimensional Space-Time: [Lec6b Download Lec6b] Concentration of Measure [notes] Download [notes] Statistical Machine Learning 2: Sampling and Learning with MCMC Variations [Lec8 Download Lec8] |
[PML2] Ch 12.1, 12.2, 12.3 12.6 |
|
Wed 02-12-2025 |
9. Statistical Machine Learning 3: Bayesian Inference with MCMC and Variational Inference [Lec9] |
[BHK] Chap 2.7 [PML2] Ch 12.1, 12.2, 12.3 12.6 |
|
Mon 02-17-2025 |
10. Stochastic Optimization Formulations and Statistical Machine Learning [notes] Learning SVM via Continuous Stochastic Gradient Descent Optimization [Lec10]
Download [Lec10] |
[PML1] Ch 8.1, 8.2, 8.3, 8.4, 8.5 [PML2] Ch 6.3 |
|
Wed 02-19-2025 |
11. Optimization and Machine Learning 1: KKT, LP, QP, SDP, SGD, [notes] |
[PML1] Ch 8.6 |
[A3 Download A3] will be out on Friday; |
Mon 02-24-2025 |
12. Non-convex Optimization: Projected Stochastic Gradient [Lec12
Download Lec12] |
[BHK] Ch 2.7 [PML1] Ch 7 |
|
Wed 02-26-2025 |
13. Stochastic Optimization - Connections of MCMC and VAE [notes] |
[PML1] Ch 11.4 [PML2] Ch 15.2.6, 28.6.5 |
|
Mon 03-03-2025 |
14: Random Projections, Johnson-Lindenstrauss, Compressive Sensing, [Lec12] Download [Lec12] |
[PML1] Ch 8.7 [PML2] Ch 6.5 |
|
Wed 03-05-2025 |
15. Matrix Sampling and Sketching [notes]
|
[PML1] Ch 17.2 [PML2] Ch 18.1,18.2,18,3, 18.5 |
|
Mon 03-10-2025 |
16. Tensor Sketching in Space-Time [notes] Download [notes] |
[PML1] Ch 4.6 [PML2] Ch 3.4, 18.4, 18.6 M1] Ch 1, 2, 5.8 |
|
Wed 03-12-2025 |
12. Hamiltonians, Symplectic Manifold and Controllable Flows I [notes] |
|
|
Mon 03-24-2025 |
17. Stochastic Hamiltonians, Symplectic Manifold and Controllable Free Energy Flows II [notes]
|
[ |
|
Wed 03-26-2025 |
MIDTERM
|
[PML1] Ch 13.3 [PML2] Ch 23.1 23.2
|
|
Mon 03-31-2025 |
19. Data Clustering with Hamiltonians and Hamiltonian Dynamics [notes] |
[M1] Ch 4, 5.7 [M2] Ch 11, 12 |
|
Wed 04-02-2025 |
[M1] Ch 2.5 [M3] Ch 13.1, 13.2, 13.3 |
[A4 Download A4] will be out; |
|
Mon 04-07-2025 |
21. Stochastic Gradient Hamiltonian MC with Controllable Hamiltonian Dynamics [notes] |
[PML2] Ch 8.1-8.5 [M1] Ch 1.3, 5.1-5.8 |
Project details will be out; [check here for final project Download check here for final project] |
Wed 04-09-2025 |
22. Reinforcement Learning 1: Learning Dynamics with Optimal Control: Dynamics LQR, iLQR, iLQG [Lec22] Download [Lec22]
|
[M3] Ch 13.4, 13.5, 13.6, 14.1-14.5, 14.13 |
|
Mon 04-14-2025 |
23. Reinforcement Learning 2: Guided Policy Search [Lec23 Download Lec23] |
[PML2] Ch 35.3, 35.4 | |
Wed 04-16-2025 |
24. Reinforcement Learning 3: Hamiltonian Dynamics, Pontryagin Maximum Principle , NeuralPMP [notes} [arxiv] Links to an external site. Also Safe Pontryagin[arxiv] |
[PML2] Ch 34.4 |
|
Mon 04-21-2025 |
25. Reinforcement Learning 4: Policy Gradient Optimization Methods [notes], Stochastic Hamiltonian Gradient Flows [notes] |
[PML2] Ch 29.3, 29.7, 29.12 [Basar] See Lectures 1, 2, 3 |
|
Wed 04-23-2025 |
26. Reinforcement Learning 5: Stochastic Hamiltonian Flows I [notes] |
[PML2] Ch 35.6 | |
Mon 04-28-2025 [Online] |
27. Reinforcement Learning 5: Forward and Inverse Problems, Scientific Discovery [notes] |
[PML2] Ch 24, 25, 34.5 |
|
|
|
||
Addtl. Material |
Probability, Information and Probabilistic Inequalities [notes Download notes] Log-Sum-Exponential-Stability [notes Download notes] |
PML1] Ch 4.1, 4.2, 4.5, 4.7, 6.1, 6.2. [PML2] Ch 3.3, 3.8, 5.1, 5.2 [PML1] Ch 3.2, 5.2 |
Some important Classical Machine Learning Background. |
Addtl. Material |
Learning by Random Walks on Graphs [notes-BHK]
Download [notes-BHK] Wasserstein Gradient Flows and the Fokker - Planck Equation[notes] .Learning Dynamics with Stochastic Processes [notes] Download [notes]
|
S
|
Important Topics on Bayesian and Reimannian Manifold Optimization and Reinforcement Learning. |
Project FAQ
1. How long should the project report be?
Answer: See directions in the Project section in assignments. For full points, please address each of the evaluation questions as succinctly as possible. You will get feedback on your presentations, which should also be incorporated into your final report.
Assignments, Exam, Final Project, and Presentation
There will be four take-home bi-weekly assignments, one in-class midterm exam, one take-home final project (in lieu of a final exam), and one presentation based on your project progress. The important deadline dates are:
- Midterm: Wednesday, March 26, 2:00 pm - 3:30 pm.
- Final Project Written Report Part 1: Due April 21st, 11:59 pm.
- Final Project Written Report, and Presentation Video, Due May 3rd, 11:59 pm
Assignments
There will be four written take-home HW assignments and one take-home final project report. Please refer to the above schedule for assignments and the final project report due time.
Assignment solutions that are turned in late shall suffer a 10% per day reduction in credit and a 100% reduction once solutions are posted.
Course Requirements and Grading
Grades will be based on these factors:
- In-class attendance and participation (5%)
- HW assignments (50% and with the potential to get extra credit)
4 assignments. Some assignments may have extra questions for extra points you can earn. (They will be specified in the assignment sheet each time.)
- In-class midterm exam (15%)
- First Presentation & Report (10%)
- Final Presentation & Report (20%)
Students with Disabilities. Students with disabilities may request appropriate academic accommodations from the Division of Diversity and Community Engagement, Services for Students with Disabilities, 471-6259, http://www.utexas.edu/diversity/ddce/ssd
Accommodations for Religious Holidays. By UT Austin policy, you must notify the instructor of your pending absence at least fourteen days prior to the date of observance of a religious holiday. If you must miss a class or an examination in order to observe a religious holiday, you will be given an opportunity to complete the missed work within a reasonable time before or after the absence, provided proper notification is given.
Statement on Scholastic Dishonesty. Anyone who violates the rules for the HW assignments or who cheats on in-class tests or the final exam is in danger of receiving an F for the course. Additional penalties may be levied by the Computer Science department, CSEM, and the University. See http://www.cs.utexas.edu/academics/conduct