Course Syllabus

This course is co-linked with CSE392(#63165) and M393C(#54664)

Instructor:

Professor Chandrajit Bajaj

NOTE: All questions related to class should be posted through Piazza. Here is the link to register for Piazza: Links to an external site. You can also join via the Piazza Tab on the Canvas course page.

Teaching Assistant

Yi Wang

Note: Please attempt to make reservations a day before to avoid conflicts. 

Course Motivation and Synopsis

This Spring and Fall course  this year is on the design and performance analysis of optimally controlled, aka reinforcement learned statistical machine learning algorithms, trained, verified  and validated on filtered, noisy observation data distributions collected from various multi-scale dynamical systems. The principal performance metrics will be on online and energy efficient training, verification and validation protocols that achieve principled and stable learning for maximal generalizability .  The emphasis will be on  possibly corrupted data and/or  the lack of full information for the learned stochastic decision making dynamic algorithmic  process. Special emphasis will also  be given to the underlying  mathematical and statistical physics principles  of  Free Energy and stochastic Hamiltonian  dyamics . Students shall  thus be exposed to the latest stochastic  machine learning  modeling approaches for  optimized decision-making, multi-player games involving stochastic dynamical systems and optimal stochastic control. These latter topics are foundational to the training of multiple neural networks (agents) both cooperatively and in adversarial scenarios to optimize the learning process of all the agents.

An initial listing of lecture topics and reference material are given in the syllabus below. This is subject to some modification, given the background and speed at which we cover ground.  Homework exercises shall be given almost bi-weekly.  Assignment solutions that are turned in late shall suffer a 10% per day reduction in credit and a 100% reduction once solutions are posted. There will be a mid-term exam in class. The exam content will be similar to the homework exercises. A list of topics will also be assigned as take-home final projects to train the best of scientific machine-learned decision-making (agents). The projects will involve modern ML programming, an oral presentation, and a written report submitted at the end of the semester.

This project shall be graded and be in lieu of a final exam.

The course is open to graduate students in all disciplines. Those in the 5-year master's program students, and in the CS, CSEM, ECE, MATH, STAT, PHYS, CHEM, and BIO, are welcome. You’ll need an undergraduate level  background  in  the intertwined topics of algorithms, data structures, numerical methods, numerical optimization,  functional analysis, algebra, geometry, topology, statistics, stochastic processes . You will need programming experience (e.g., Python ), at  a CS undergraduate senior level.

 

Course Reference Material (+ reference papers cited in lectures )

  1. [B1] Chandrajit Bajaj (frequently updated)  A Mathematical Primer for Computational Data Sciences 
  2. [PML1] Kevin Murphy Probabilistic Machine Learning: An Introduction. Download Probabilistic Machine Learning: An Introduction.
  3. [PML2] Kevin Murphy Probabilistic Machine Learning: Advanced Topics. Download Probabilistic Machine Learning: Advanced Topics.
  4. [M1] Peter S. Maybeck Stochastic Models, Estimation and Control Volume 1 Download Stochastic Models, Estimation and Control Volume 1
  5. [M2] Peter S. Maybeck Stochastic Models, Estimation and Control Volume 2 Download Stochastic Models, Estimation and Control Volume 2
  6. [M3] Peter S. Maybeck Stochastic Models, Estimation and Control Volume 3 Download Stochastic Models, Estimation and Control Volume 3
  7. [MU] Michael Mitzenmacher, Eli Upfal Probability and Computing (Randomized Algorithms and Probabilistic Analysis) Download Probability and Computing (Randomized Algorithms and Probabilistic Analysis)
  8. [SB] Richard Sutton, Andrew Barto Reinforcement Learning
  9. [SD] Shai Shalev-Shwartz, Shai Ben-David Understanding Machine Learning, From Theory to Algorithms
  10. [Basar] Tamer Basar  Lecture Notes on Non-Cooperative Game Theory.
  11. [BHK] Avrim Blum, John Hopcroft, and Ravindran Kannan. Foundations of Data Science
  12. [BV] Stephen Boyd and Lieven Vandenberghe Convex Optimization.
  13. Extra reference materials.

TENTATIVE  COURSE OUTLINE (in Flux). 

Date Topic Reading Assignments

Mon

01-13-2025

1. Introduction to High-Dimensional Spaces, Belief, and Decision-Making Spaces, [Lec1 Download Lec1]

Dynamical Systems and Deep Learning [notes] Download [notes]

 

[PML1] Ch 1.1, 1.2, 1.3, 1.4

 

Wed

01-15-2025

2. Stochastic Machine Learning: Entropy, Distributional Estimates [notes] Download notes]

Geometry of Norms and Approximations [notes Download notes];

[PML1] Ch 2.1, 2.2, 2.3, 2.4, 2.5, 2.6

[PML1] Ch 4.1, 4.2, 4.5, 4.7, 6.1, 6.2.

[A1 Download A1] with [latex template Download latex template] out today; 

Fri

01-17-2025

(GDC 4.302)

Zoom Record Links to an external site.

3. Multivariate Gaussians, Infinite Dimensional Spaces  and Gaussian Processes [Notes] [Lec3] Download [Lec3]

Introduction to Stochastic Processes [Supp Notes] Download [Supp Notes]

 

 

[M1] Ch 3

[PML1] Ch 17.2

[PML2] Ch 18.1,18.2,18,3, 18.5

Mon

01-27-2025

4.  Additional notes for Gaussian Processes [notes Download notes]

 Bayesian Deep Learning [notes]

Univariate Time Series Analysis [supp notes]

 

[PML1] Ch 17.2

[PML2] Ch 18.1,18.2,18,3, 18.5

 

 

Wed

01-29-2025

5. Time Series Bayesian Regression with Deep Gaussian Processes,   Kalman Filtering [Supp notes][Supp Notes2]

Basics of Kalman Filters [Lec5 Download Lec5]

Illustrated Kalman Filters [notes Download notes]

 

 

 

[A2 Download A2] will be out on Friday;

Mon

02-03-2025

6.  MonteCarlo vs Quasi Monte-Carlo Sampling: Relationship to Integration Error H-K Inequality [Lec7 Download Lec7][notes Download notes]

 

[BHK] Ch 2.1-2.6

[PML2] Ch 11

 

Wed

02-05-2025

7.  Statistical Machine Learning 2: Introduction to Markov Chains, Page Rank, MCMC [Lec7]

[PML1] Ch 3.6

[PML2] Ch 2.6, 4.2, 7.4.5, 7.4.6,

 

Mon

02-10-2025

8.  Sampling in High-Dimensional  Space-Time:    [Lec6b Download Lec6b]  Concentration of Measure  [notes] Download [notes]

Statistical Machine Learning 2: Sampling and Learning  with MCMC Variations [Lec8 Download Lec8]

[PML2] Ch 12.1, 12.2, 12.3 12.6

Wed

02-12-2025

9. Statistical Machine Learning 3:  Bayesian Inference with MCMC and Variational Inference [Lec9]

[BHK] Chap 2.7

[PML2] Ch 12.1, 12.2, 12.3 12.6

 

Mon

02-17-2025

10.  Stochastic Optimization Formulations and Statistical Machine Learning [notes]

Learning SVM via Continuous Stochastic Gradient Descent Optimization [Lec10] Download [Lec10]

Continuous Stochastic (Noisy) Gradient Descent (SGD)  -- Simulated Annealing, Fokker-Planck [notes Download notes]

[PML1] Ch 8.1, 8.2, 8.3, 8.4, 8.5

[PML2] Ch 6.3

Wed

02-19-2025

11. Optimization and Machine Learning 1: KKT, LP, QP, SDP, SGD, [notes]

[PML1] Ch 8.6

[A3 Download A3] will be out on Friday;

Mon

02-24-2025

 

12.  Non-convex Optimization: Projected Stochastic Gradient [Lec12 Download Lec12]

Learning with  SGD variations, Adagrad, RMSProp, Adam, ...] [supp-notes] Download [supp-notes]

[BHK] Ch 2.7

[PML1] Ch 7

 

Wed

02-26-2025

13.   Stochastic Optimization - Connections of  MCMC and VAE [notes]

[PML1] Ch 11.4

[PML2] Ch 15.2.6, 28.6.5

 

Mon

03-03-2025

14: Random Projections, Johnson-Lindenstrauss, Compressive Sensing, [Lec12] Download [Lec12] 

[PML1] Ch 8.7

[PML2] Ch 6.5

 

 

Wed

03-05-2025

 15.  Matrix Sampling and Sketching [notes]

 

[PML1] Ch 17.2

[PML2] Ch 18.1,18.2,18,3, 18.5

 

Mon

03-10-2025

16.   Tensor Sketching  in Space-Time [notes] Download [notes]

[PML1] Ch 4.6

[PML2] Ch 3.4, 18.4, 18.6

M1] Ch 1, 2, 5.8

Wed

03-12-2025

 

12.  Hamiltonians, Symplectic Manifold and Controllable Flows I  [notes]


 

 

Mon

03-24-2025

17.  Stochastic Hamiltonians, Symplectic Manifold and Controllable  Free Energy  Flows II  [notes]

 

[

 

Wed

03-26-2025

 MIDTERM

 

[PML1] Ch 13.3

[PML2] Ch 23.1 23.2

 

 

Mon

03-31-2025

19.  Data Clustering with Hamiltonians and Hamiltonian Dynamics [notes]

[M1] Ch 4, 5.7

[M2] Ch 11, 12

 

 

Wed

04-02-2025

20.  Learning Dynamics with Control and Optimality [ notes]

[M1] Ch 2.5

[M3] Ch 13.1, 13.2, 13.3

[A4 Download A4] will be out;

Mon

04-07-2025

21. Stochastic Gradient Hamiltonian MC with Controllable Hamiltonian Dynamics [notes]

[PML2] Ch 8.1-8.5

[M1] Ch 1.3, 5.1-5.8

Project details will be out; [check here for final project Download check here for final project]

Wed

04-09-2025

22.  Reinforcement Learning 1: Learning Dynamics with Optimal Control:  Dynamics  LQR, iLQR, iLQG [Lec22] Download [Lec22] 

 

[M3] Ch 13.4, 13.5, 13.6, 14.1-14.5, 14.13

 

 

 

Mon

04-14-2025

 23. Reinforcement Learning 2: Guided Policy Search [Lec23 Download Lec23]

[PML2] Ch 35.3, 35.4

Wed

04-16-2025

24.   Reinforcement Learning 3:   Hamiltonian Dynamics, Pontryagin Maximum Principle , NeuralPMP [notes} [arxiv]  Links to an external site. Also Safe Pontryagin[arxiv]

[PML2] Ch 34.4

 

Mon

04-21-2025

25.  Reinforcement Learning 4:  Policy Gradient Optimization Methods  [notes],  Stochastic Hamiltonian Gradient Flows [notes]

[PML2] Ch 29.3, 29.7, 29.12

[Basar] See Lectures 1, 2, 3 

 

 

Wed

04-23-2025

26. Reinforcement Learning 5: Stochastic Hamiltonian Flows I [notes]

[PML2] Ch 35.6

Mon

04-28-2025

[Online]

27.  Reinforcement Learning 5: Forward and Inverse Problems, Scientific Discovery [notes]

[PML2] Ch 24, 25, 34.5

 

 

Addtl. Material

Probability, Information and Probabilistic Inequalities [notes Download notes]

Log-Sum-Exponential-Stability [notes Download notes]

PML1] Ch 4.1, 4.2, 4.5, 4.7, 6.1, 6.2.

[PML2] Ch 3.3, 3.8, 5.1, 5.2

[PML1] Ch 3.2, 5.2

Some important Classical Machine Learning Background.

Addtl. Material

Learning by Random Walks on Graphs  [notes-BHK] Download [notes-BHK]
The Markov-chain Monte Carlo Interactive Gallery Links to an external site.

Wasserstein Gradient Flows and the Fokker - Planck Equation[notes]

.Learning Dynamics with Stochastic Processes [notes] Download [notes] 

 

S

 

Important Topics on Bayesian and Reimannian Manifold Optimization and Reinforcement Learning.

 

Project FAQ

1. How long should the project report be?

Answer: See directions in the Project section in assignments. For full points, please address each of the evaluation questions as succinctly as possible. You will get feedback on your presentations, which should also be incorporated into your final report.

Assignments, Exam, Final Project, and Presentation

There will be four take-home bi-weekly assignments, one in-class midterm exam, one take-home final project (in lieu of a final exam), and one presentation based on your project progress. The important deadline dates are:

  • Midterm: Wednesday, March 26, 2:00 pm - 3:30 pm.
  • Final Project Written Report Part 1: Due April 21st, 11:59 pm.
  • Final Project Written Report, and Presentation Video, Due May 3rd, 11:59 pm

Assignments

There will be four written take-home HW assignments and one take-home final project report. Please refer to the above schedule for assignments and the final project report due time.

Assignment solutions that are turned in late shall suffer a 10% per day reduction in credit and a 100% reduction once solutions are posted.

Course Requirements and Grading

Grades will be based on these factors:

  • In-class attendance and participation (5%)
  • HW assignments (50% and with the potential to get extra credit) 

4 assignments. Some assignments may have extra questions for extra points you can earn. (They will be specified in the assignment sheet each time.)

  • In-class midterm exam (15%) 
  • First Presentation & Report (10%)
  • Final Presentation & Report (20%)  

Students with Disabilities. Students with disabilities may request appropriate academic accommodations from the Division of Diversity and Community Engagement, Services for Students with Disabilities, 471-6259, http://www.utexas.edu/diversity/ddce/ssd . 

 

Accommodations for Religious Holidays. By UT Austin policy, you must notify the instructor of your pending absence at least fourteen days prior to the date of observance of a religious holiday. If you must miss a class or an examination in order to observe a religious holiday, you will be given an opportunity to complete the missed work within a reasonable time before or after the absence, provided proper notification is given.

 

Statement on Scholastic Dishonesty. Anyone who violates the rules for the HW assignments or who cheats on in-class tests or the final exam is in danger of receiving an F for the course. Additional penalties may be levied by the Computer Science department, CSEM, and the University. See http://www.cs.utexas.edu/academics/conduct