# Distributed Optimization and Machine Learning

- Institution: IIT Bombay
- Department: Computer Science and Engineering
- Teacher: Prof. Mayank Baranwal
- Level: Graduate
- Language: English
- Subjects: Computer science, Engineering
- Source: [NPTEL](https://nptel.ac.in/courses/106101466)
- Licence: [CC BY-NC-SA](https://nptel.ac.in/faq) (Non-commercial use only.)
- Syllabus: [at the source](https://nptel.ac.in/courses/106101466)
- Who can watch: Tweens profiles and older (sources give no ratings; placed by Invisible Network from the source's own words)

Centralized access to information and its subsequent processing is often computationally prohibitive over large networks due to communication overhead and the scale of the problem. Consequently, such systems rely on control and optimization algorithms that are fully distributed or even decentralized in nature. This course will provide a comprehensive overview of design and analysis of distributed optimization algorithms and their applications to machine learning. The aim is to revisit classical control and optimization algorithms for centralized optimization and discuss how these can be extended to distributed setting to accommodate the effects of communication constraints, network topology, computational resources, and robustness. Topics include graph theory, iterative methods for convex problems, synchronous and asynchronous setups, consensus algorithms, and distributed machine learning. We will also explore some recent literature in this area that exploits control theory for design of accelerated distributed optimization algorithms.

Intended audience: MTech and PhD students in broad areas of optimization and data science

Prerequisites: A background in convex optimization and differential equations is preferred

## Lectures

1. [Lecture 1: Introduction to optimization](https://invisiblenetwork.ai/learn/nptel/106101466/03e187cc7eda) (YouTube)
2. [Lecture 2: Analyzing optimization algorithms in continuous time domain](https://invisiblenetwork.ai/learn/nptel/106101466/26a047e4e3d3) (YouTube)
3. [Lecture 3: Course Outline](https://invisiblenetwork.ai/learn/nptel/106101466/9f30b16e7e0e) (YouTube)
4. [Lecture 4: Basics of optimization problems](https://invisiblenetwork.ai/learn/nptel/106101466/e07191871072) (YouTube)
5. [Lecture 5: Convex sets and Convex functions](https://invisiblenetwork.ai/learn/nptel/106101466/3ff08296e213) (YouTube)
6. [Lecture 6: Strictly and strongly convex functions](https://invisiblenetwork.ai/learn/nptel/106101466/11b6bfa5f552) (YouTube)
7. [Lecture 7: Implications of strong convexity](https://invisiblenetwork.ai/learn/nptel/106101466/164f1ec5dcfb) (YouTube)
8. [Lecture 8: Primal and dual optimization problems](https://invisiblenetwork.ai/learn/nptel/106101466/357eafbb1cd1) (YouTube)
9. [Lecture 9: Slaters condition](https://invisiblenetwork.ai/learn/nptel/106101466/c9a88507063b) (YouTube)
10. [Lecture 10: Analysis of gradient descent algorithm](https://invisiblenetwork.ai/learn/nptel/106101466/22d592659e6a) (YouTube)
11. [Lecture 11: KKT conditions](https://invisiblenetwork.ai/learn/nptel/106101466/3b8c79d14e5f) (YouTube)
12. [Lecture 12: Acceleration under strong convexity](https://invisiblenetwork.ai/learn/nptel/106101466/30b440e47721) (YouTube)
13. [Lecture 13: Accelerate the convergence even further](https://invisiblenetwork.ai/learn/nptel/106101466/0c634513372d) (YouTube)
14. [Lecture 14: Stability theory](https://invisiblenetwork.ai/learn/nptel/106101466/3c847fb76c8c) (YouTube)
15. [Lecture 15: Connections to optimization problems](https://invisiblenetwork.ai/learn/nptel/106101466/2e3167d3e5bd) (YouTube)
16. [Lecture 16: Exponential stability](https://invisiblenetwork.ai/learn/nptel/106101466/17dfbdf6a71b) (YouTube)
17. [Lecture 17: Bregman Divergance](https://invisiblenetwork.ai/learn/nptel/106101466/cf3c4475fac0) (YouTube)
18. [Lecture 18: Rescaled Gradient Flow](https://invisiblenetwork.ai/learn/nptel/106101466/67d5274dc816) (YouTube)
19. [Lecture 19: Advanced Results on PL inequality: Part 1](https://invisiblenetwork.ai/learn/nptel/106101466/e1376059a372) (YouTube)
20. [Lecture 20: Advanced Results on PL inequality: Part 2](https://invisiblenetwork.ai/learn/nptel/106101466/c496652f1f74) (YouTube)
21. [Lecture 21: Constrained Optimization Problem](https://invisiblenetwork.ai/learn/nptel/106101466/2b828236b02a) (YouTube)
22. [Lecture 22: Augmented Lagrangian](https://invisiblenetwork.ai/learn/nptel/106101466/a544e5a4dc7e) (YouTube)
23. [Lecture 23: Method of Multipliers](https://invisiblenetwork.ai/learn/nptel/106101466/25709eab2d7a) (YouTube)
24. [Lecture 24: Dual Ascent and Dual Decomposition](https://invisiblenetwork.ai/learn/nptel/106101466/1c3b7c69f0a3) (YouTube)
25. [Lecture 25: ADMM Algorithm](https://invisiblenetwork.ai/learn/nptel/106101466/f0cc60be5c74) (YouTube)
26. [Lecture 26: Basics of Graph Theory](https://invisiblenetwork.ai/learn/nptel/106101466/751c8e07cb81) (YouTube)
27. [Lecture 27: Basics of Graph Theory-2](https://invisiblenetwork.ai/learn/nptel/106101466/c0f0284bc5cf) (YouTube)
28. [Lecture 28: Consensus and Average Consensus](https://invisiblenetwork.ai/learn/nptel/106101466/c1b14071cba7) (YouTube)
29. [Lecture 29: Consensus and Average Consensus-2](https://invisiblenetwork.ai/learn/nptel/106101466/309cbefb9672) (YouTube)
30. [Lecture 30: Consensus Algorithms](https://invisiblenetwork.ai/learn/nptel/106101466/2af83d70d444) (YouTube)
31. [Lecture 31: Consensus Algorithms-Fixed time](https://invisiblenetwork.ai/learn/nptel/106101466/ce17c6bf2220) (YouTube)
32. [Lecture 32: Distributed Economic Dispatch Problem](https://invisiblenetwork.ai/learn/nptel/106101466/d490ed2c4c1f) (YouTube)
33. [Lecture 33: Algorithm for Uncapacitated EDP](https://invisiblenetwork.ai/learn/nptel/106101466/6582172d22d7) (YouTube)
34. [Lecture 34: Capacitated EDP](https://invisiblenetwork.ai/learn/nptel/106101466/0e5fa2d403c2) (YouTube)
35. [Lecture 35: Algorithms for Distributed Optimization](https://invisiblenetwork.ai/learn/nptel/106101466/8a5b9ac857b7) (YouTube)
36. [Lecture 36: Algorithms for Distributed Optimization-2](https://invisiblenetwork.ai/learn/nptel/106101466/0f2544b8caa1) (YouTube)
37. [Lecture 37: Continuous-time Distributed Optimization Algorithms](https://invisiblenetwork.ai/learn/nptel/106101466/4034842f5da4) (YouTube)
38. [Lecture 38: Introduction to Neural Networks](https://invisiblenetwork.ai/learn/nptel/106101466/d83121710dd2) (YouTube)
39. [Lecture 39: Large Scale Machine Learning](https://invisiblenetwork.ai/learn/nptel/106101466/08bbfcaf8bac) (YouTube)
40. [Lecture 40: Decentralized Stochastic Gradient Descent](https://invisiblenetwork.ai/learn/nptel/106101466/2442e7776d2f) (YouTube)
41. [Lecture 41: Decentralized Stochastic Gradient Descent -2](https://invisiblenetwork.ai/learn/nptel/106101466/948d295e849e) (YouTube)
42. [Lecture 42: Introduction to Federated Learning](https://invisiblenetwork.ai/learn/nptel/106101466/283373168fb0) (YouTube)
43. [Lecture 43: FedAvg Algorithm](https://invisiblenetwork.ai/learn/nptel/106101466/5b0b1ac49299) (YouTube)
44. [Lecture 44: Convergence Analysis of FL](https://invisiblenetwork.ai/learn/nptel/106101466/58c5dc870892) (YouTube)
45. [Lecture 45: Sources of Computational Heterogenity in FL](https://invisiblenetwork.ai/learn/nptel/106101466/38aec48f580b) (YouTube)
46. [Lecture 46: Objective Inconsistency Problem](https://invisiblenetwork.ai/learn/nptel/106101466/44ffc31d6f2f) (YouTube)
47. [Lecture 47: General Update Rule](https://invisiblenetwork.ai/learn/nptel/106101466/ad7348e87ef1) (YouTube)