Course106106184117 lectures
Deep Learning - IIT Ropar
Prof. Sudarshan Iyengar · IIT Ropar
About
Deep Learning has received a lot of attention over the past few years and has been employed successfully by companies like Google, Microsoft, IBM, Facebook, Twitter etc. to solve a wide range of problems in Computer Vision and Natural Language Processing. In this course we will learn about the building blocks used in these Deep Learning based solutions. Specifically, we will learn about feedforward neural networks, convolutional neural networks, recurrent neural networks and attention mechanisms. We will also look at various optimization algorithms such as Gradient Descent, Nesterov Accelerated Gradient Descent, Adam, AdaGrad and RMSProp which are used for training such deep neural networks. At the end of this course students would have knowledge of deep architectures used for solving various Vision and NLP tasks
Intended audience: Any Interested Learners
Prerequisites: Working knowledge of Linear Algebra, Probability Theory. It would be beneficial if the participants have done a course on Machine Learning.
Lectures
Week 1
- Biological Neuron
- From Spring to Winter of AI
- The Deep Revival
- From Cats to Convolutional Neural Networks
- Faster, higher, stronger
- The Curious Case of Sequences
- Beating humans at their own games (literally)
- The Madness (2013-)
- (Need for) Sanity
- Motivation from Biological Neurons
- McCulloch Pitts Neuron, Thresholding Logic
- Perceptrons
- Error and Error Surfaces
- Perceptron Learning Algorithm
- Proof of Convergence of Perceptron Learning Algorithm
Week 2
- Linearly Separable Boolean Functions
- Representation Power of a Network of Perceptrons
- Sigmoid Neuron
- A typical Supervised Machine Learning Setup
- Learning Parameters: (Infeasible) guess work
- Learning Parameters: Gradient Descent
- Representation Power of Multilayer Network of Sigmoid Neurons
Week 3
- Feedforward Neural Networks (a.k.a multilayered network of neurons)
- Learning Paramters of Feedforward Neural Networks (Intuition)
- Output functions and Loss functions
- Backpropagation (Intuition)
- Backpropagation: Computing Gradients w.r.t. the Output Units
- Backpropagation: Computing Gradients w.r.t. Hidden Units
- Backpropagation: Computing Gradients w.r.t. Parameters
- Backpropagation: Pseudo code
- Derivative of the activation function
- Information content, Entropy & cross entropy
week 4
- Recap: Learning Parameters: Guess Work, Gradient Descent
- Contours Maps
- Momentum based Gradient Descent
- Nesterov Accelerated Gradient Descent
- Stochastic And Mini-Batch Gradient Descent
- Tips for Adjusting Learning Rate and Momentum
- Line Search
- Gradient Descent with Adaptive Learning Rate
- Bias Correction in Adam
Week 5
- Eigenvalues and Eigenvectors
- Linear Algebra : Basic Definitions
- Eigenvalue Decompositon
- Principal Component Analysis and its Interpretations
- PCA : Interpretation 2
- PCA : Interpretation 3
- PCA : Interpretation 3 (Contd.)
- PCA : Practical Example
- Singular Value Decomposition
Week 6
- Introduction to Autoncoders
- Link between PCA and Autoencoders
- Regularization in autoencoders (Motivation)
- Denoising Autoencoders
- Sparse Autoencoders
- Contractive Autoencoders
Week 7
- Bias and Variance
- Train error vs Test error
- Train error vs Test error (Recap)
- True error and Model complexity
- L2 regularization
- Dataset augmentation
- Parameter sharing and tying
- Adding Noise to the inputs
- Adding Noise to the outputs
- Early stopping
- Ensemble Methods
- Dropout
Week 8
- A quick recap of training deep neural networks
- Unsupervised pre-training
- Better activation functions
- Better initialization strategies
- Batch Normalization
Week 9
- One-hot representations of words
- Distributed Representations of words
- SVD for learning word representations
- SVD for learning word representations (Contd.)
- Continuous bag of words model
- Skip-gram model
- Skip-gram model (Contd.)
- Contrastive estimation
- Hierarchical softmax
- GloVe representations
- Evaluating word representations
- Relation between SVD and Word2Vec
week 10
- The convolution operation
- Relation between input size, output size and filter size
- Convolutional Neural Networks
- Convolutional Neural Networks (Contd.)
- CNNs (success stories on ImageNet)
- CNNs (success stories on ImageNet) (Contd.)
- Image Classification continued (GoogLeNet and ResNet)
- Visualizing patches which maximally activate a neuron
- Visualizing filters of a CNN
- Occlusion experiments
- Finding influence of input pixels using backpropagation
- Guided Backpropagation
- Optimization over images
- Create images from embeddings
- Deep Dream
- Deep Art
- Fooling Deep Convolutional Neural Networks
Week 11
- Sequence Learning Problems
- Recurrent Neural Networks
- Backpropagation through time
- The problem of Exploding and Vanishing Gradients
- Some Gory Details
- Selective Read, Selective Write, Selective Forget - The Whiteboard Analogy
- Long Short Term Memory(LSTM) and Gated Recurrent Units(GRUs)
- How LSTMs avoid the problem of vanishing gradients
- How LSTMs avoid the problem of vanishing gradients (Contd.)
Week 12
Syllabus and materials
At NPTEL, as the course publishes them.
Credits and licence
- Institution
- IIT Ropar
- Department
- Computer Science and Engineering
- Teacher
- Prof. Sudarshan Iyengar
- Level
- Undergraduate
- Language
- English
- Subjects
- Computer science, Engineering
- Source
- NPTEL
- Licence
- CC BY-NC-SA, as NPTEL states it. Non-commercial use only. NPTEL's terms
- Attribution
- NPTEL states: "Distributed under Creative Commons Attribution-ShareAlike / CC BY - NC - SA", beside a BY-NC-SA badge. Coordinated by IIT Madras.
- Who can watch
- NPTEL gives no rating. Invisible Network shows it to Tweens profiles and older, judged from what the source says about it. How courses are placed
More like this
NPTEL103106570
Course
Interfacial Engineering
40 lectures · Video
NPTEL106106169
Course
Social Networks
166 lectures · Source's player
NPTEL106106183
Course
Discrete Mathematics
400 lectures · Source's player
NPTEL108106177
Course
Digital System Design
68 lectures · Video
NPTEL106106182
Course
The Joy of Computing using Python
225 lectures · Source's player
TED
Talk
The future we're building -- and boring
Source's player
TED
Talk
The thrilling potential of SixthSense technology
Source's player
TED
Talk
Why don't perpetual motion machines ever work?
Source's player
Course
History 7b Spring 2006 UC Berkeley (History 7B US History: from Civil War to Present)
39 lectures · Audio
Course
Science Events Video
84 lectures · Video
Course
Nuclear Engineering 101 001 Fall 2009 UC Berkeley Nuclear Reactions and Radiation (Nuclear Engineering 101, 001|Fall 2009|UC Berkeley (Nuclear Reactions and Radiation))
29 lectures · Audio
Course
Matrix Methods in Data Analysis, Signal Processing, and Machine Learning
34 lectures · Video
Course
Matrix Methods in Data Analysis, Signal Processing, and Machine Learning
34 lectures · Video