Course10610600161 lectures
Foundations of Machine Learning
Prof. J. SakethaNath · IIT Hyderabad
About
This is a course on Machine Learning that introduces the subject in a formal way with a focus on basic and foundational concepts. A holistic view of machine learning is taken, including aspects of optimization (SGD) that are ignored in classical narratives. Topics that are relevant to modern models are highlighted so that the connection to state-of-the-art is not missed in the theoretical exploration. For example, in case of linear regression, apart from classical topics like bias-variance trade-off and regularization, an analysis in the overparametrization regime is also taught that immediately connects you to modern day complex models with billions of parameters. Finally, foundational topics for generative models are also covered, including energy-based models and Langevin sampling.
Intended audience: Students wishing to start research in core machine learning areas
Prerequisites: Anyone with a good understanding of basic engineering math is eligible. Especially subjects like linear algebra, probability theory, and optimization.
Lectures
Week 1
- What is ML ?
- Stochastic framework for Supervised Learning
- Risk Minimization and Bayes optimal
- Stochastic Optimization and ERM
Week 2
- Linear Regression
- Linear Regression - Estimation Error
- Supervised Learning >> Denoising/Curve-fitting
- Linear modelling
Week 3
- Linear Classification
- Logistic Regression
- Logistic Regression - ERM
- Linear Classification - Estimation Error - Part 1
- Linear Classification - Estimation Error - Part 2
Week 4
- Gradient Descent - Primal & Dual view
- Smooth Functions and examples
- Gradient Descent - Convergence to stationary point
- Rapid Convergence to global optima in Linear Regression
- KL condition and convergence
- Fast Convergence to global optima in Logistic Regression
Week 5
- Stochastic Gradient Descent (SGD)
- SGD vs GD for Linear Regression
- SGD vs GD for Logistic Regression
- Model Selection
Week 6
- What is Regularization ?
- Linear algebra motivation for min-norm regularization
- Regularization for controlling effective dimension
- Bias-variance trade-off for estimation error
- Equivalent forms of regularization
- Regularization in linear classification
- Implicit regularization with GD/SGD
Week 7
- Linear modelling in infinite-dimensional spaces
- Introduction to kernels
- Linear kernels
- Polynomial Kernels
- Few properties of kernels
- Characterization of kernels
Week 8
- Universal kernels
- Universal kernel machines
- Bayes consistency with universal kernel m/c
- Rate of generalization with kernel m/c
Week 9
- Overparametrization
- Modern perspective of ML: Minimal vs optimal regularization
- The case for latent variable modelling
- Modularity in modelling: Deep learning
- Modelling with Deep learning
Week 10
- Estimation with Deep learning
- Backpropagation-1
- Backpropagation-2
- Optimization error in 1-hidden layer nets
- Implicit bias in 1-hidden layer nets
- Optimization error in deep learning
Week 11
- Likelihood estimation
- Energy based models, MLE
- MLE examples
- Generative regression
- Generative classification
- Discriminative Models and MCLE
Week 12
Syllabus and materials
At NPTEL, as the course publishes them.
Credits and licence
- Institution
- IIT Hyderabad
- Department
- Computer Science and Engineering
- Teacher
- Prof. J. SakethaNath
- Level
- Undergraduate
- Language
- English
- Subjects
- Computer science, Engineering
- Source
- NPTEL
- Licence
- CC BY-NC-SA, as NPTEL states it. Non-commercial use only. NPTEL's terms
- Attribution
- NPTEL states: "Distributed under Creative Commons Attribution-ShareAlike / CC BY - NC - SA", beside a BY-NC-SA badge. Coordinated by IIT Madras.
- Who can watch
- NPTEL gives no rating. Invisible Network shows it to Tweens profiles and older, judged from what the source says about it. How courses are placed