Learn · MIT OpenCourseWare

The text "So, you think you know how to take derivatives?" surrounded by derivatives.

Course18.S09617 lectures

Matrix Calculus for Machine Learning and Beyond

Prof. Alan Edelman and Prof. Steven G. Johnson · Massachusetts Institute of Technology

Start with lecture 1 At MIT OpenCourseWare

About

We all know that calculus courses such as 18.01 Single Variable Calculus and 18.02 Multivariable Calculus cover univariate and vector calculus, respectively. Modern applications such as machine learning and large-scale optimization require the next big step, “matrix calculus” and calculus on arbitrary vector spaces.

This class covers a coherent approach to matrix calculus showing techniques that allow you to think of a matrix holistically (not just as an array of scalars), generalize and compute derivatives of important matrix factorizations and many other complicated-looking operations, and understand how differentiation formulas must be reimagined in large-scale computing.

Lectures

Lecture Videos

  1. Lecture 1 Part 1: Introduction and Motivation Video
  2. Lecture 1 Part 2: Derivatives as Linear Operators Video
  3. Lecture 2 Part 1: Derivatives in Higher Dimensions: Jacobians and Matrix Functions Video
  4. Lecture 2 Part 2: Vectorization of Matrix Functions Video
  5. Lecture 3 Part 1: Kronecker Products and Jacobians Video
  6. Lecture 3 Part 2: Finite-Difference Approximations Video
  7. Lecture 4 Part 1: Gradients and Inner Products in Other Vector Spaces Video
  8. Lecture 4 Part 2: Nonlinear Root Finding, Optimization, and Adjoint Gradient Methods Video
  9. Lecture 5 Part 1: Derivative of Matrix Determinant and Inverse Video
  10. Lecture 5 Part 2: Forward Automatic Differentiation via Dual Numbers Video
  11. Lecture 5 Part 3: Differentiation on Computational Graphs Video
  12. Lecture 6 Part 1: Adjoint Differentiation of ODE Solutions Video
  13. Lecture 6 Part 2: Calculus of Variations and Gradients of Functionals Video
  14. Lecture 7 Part 1: Derivatives of Random Functions Video
  15. Lecture 7 Part 2: Second Derivatives, Bilinear Forms, and Hessian Matrices Video
  16. Lecture 8 Part 1: Derivatives of Eigenproblems Video
  17. Lecture 8 Part 2: Automatic Differentiation on Computational Graphs Video

Syllabus and materials

At MIT OpenCourseWare, as the course publishes them.

Credits and licence

Institution
Massachusetts Institute of Technology
Department
Mathematics
Taught by
Prof. Alan Edelman, Prof. Steven G. Johnson
Term
January IAP 2023
Level
Undergraduate
Language
English
Subjects
Mathematics
Source
MIT OpenCourseWare
Licence
CC BY-NC-SA 4.0, as MIT OpenCourseWare states it. Non-commercial use only. MIT OpenCourseWare's terms
Attribution
Prof. Alan Edelman, Prof. Steven G. Johnson. Matrix Calculus for Machine Learning and Beyond. January IAP 2023. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu/. License: Creative Commons BY-NC-SA.
Who can watch
MIT OpenCourseWare gives no rating. Invisible Network shows it to Tweens profiles and older, judged from what the source says about it. How courses are placed