Learn · Deep Learning - IIT Ropar
Lecture 10 of 117Week 1
Motivation from Biological Neurons
Prof. Sudarshan Iyengar · IIT Ropar
Credits and licence
- From
- Deep Learning - IIT Ropar, IIT Ropar (at NPTEL)
- Licence
- CC BY-NC-SA, as NPTEL states it. Non-commercial use only.
- Attribution
- NPTEL states: "Distributed under Creative Commons Attribution-ShareAlike / CC BY - NC - SA", beside a BY-NC-SA badge. Coordinated by IIT Madras.
- Who can watch
- Tweens profiles and older. How courses are placed
All the lectures
- Biological Neuron
- From Spring to Winter of AI
- The Deep Revival
- From Cats to Convolutional Neural Networks
- Faster, higher, stronger
- The Curious Case of Sequences
- Beating humans at their own games (literally)
- The Madness (2013-)
- (Need for) Sanity
- Motivation from Biological Neurons
- McCulloch Pitts Neuron, Thresholding Logic
- Perceptrons
- Error and Error Surfaces
- Perceptron Learning Algorithm
- Proof of Convergence of Perceptron Learning Algorithm
- Linearly Separable Boolean Functions
- Representation Power of a Network of Perceptrons
- Sigmoid Neuron
- A typical Supervised Machine Learning Setup
- Learning Parameters: (Infeasible) guess work
- Learning Parameters: Gradient Descent
- Representation Power of Multilayer Network of Sigmoid Neurons
- Feedforward Neural Networks (a.k.a multilayered network of neurons)
- Learning Paramters of Feedforward Neural Networks (Intuition)
- Output functions and Loss functions
- Backpropagation (Intuition)
- Backpropagation: Computing Gradients w.r.t. the Output Units
- Backpropagation: Computing Gradients w.r.t. Hidden Units
- Backpropagation: Computing Gradients w.r.t. Parameters
- Backpropagation: Pseudo code
- Derivative of the activation function
- Information content, Entropy & cross entropy
- Recap: Learning Parameters: Guess Work, Gradient Descent
- Contours Maps
- Momentum based Gradient Descent
- Nesterov Accelerated Gradient Descent
- Stochastic And Mini-Batch Gradient Descent
- Tips for Adjusting Learning Rate and Momentum
- Line Search
- Gradient Descent with Adaptive Learning Rate
- Bias Correction in Adam
- Eigenvalues and Eigenvectors
- Linear Algebra : Basic Definitions
- Eigenvalue Decompositon
- Principal Component Analysis and its Interpretations
- PCA : Interpretation 2
- PCA : Interpretation 3
- PCA : Interpretation 3 (Contd.)
- PCA : Practical Example
- Singular Value Decomposition
- Introduction to Autoncoders
- Link between PCA and Autoencoders
- Regularization in autoencoders (Motivation)
- Denoising Autoencoders
- Sparse Autoencoders
- Contractive Autoencoders
- Bias and Variance
- Train error vs Test error
- Train error vs Test error (Recap)
- True error and Model complexity
- L2 regularization
- Dataset augmentation
- Parameter sharing and tying
- Adding Noise to the inputs
- Adding Noise to the outputs
- Early stopping
- Ensemble Methods
- Dropout
- A quick recap of training deep neural networks
- Unsupervised pre-training
- Better activation functions
- Better initialization strategies
- Batch Normalization
- One-hot representations of words
- Distributed Representations of words
- SVD for learning word representations
- SVD for learning word representations (Contd.)
- Continuous bag of words model
- Skip-gram model
- Skip-gram model (Contd.)
- Contrastive estimation
- Hierarchical softmax
- GloVe representations
- Evaluating word representations
- Relation between SVD and Word2Vec
- The convolution operation
- Relation between input size, output size and filter size
- Convolutional Neural Networks
- Convolutional Neural Networks (Contd.)
- CNNs (success stories on ImageNet)
- CNNs (success stories on ImageNet) (Contd.)
- Image Classification continued (GoogLeNet and ResNet)
- Visualizing patches which maximally activate a neuron
- Visualizing filters of a CNN
- Occlusion experiments
- Finding influence of input pixels using backpropagation
- Guided Backpropagation
- Optimization over images
- Create images from embeddings
- Deep Dream
- Deep Art
- Fooling Deep Convolutional Neural Networks
- Sequence Learning Problems
- Recurrent Neural Networks
- Backpropagation through time
- The problem of Exploding and Vanishing Gradients
- Some Gory Details
- Selective Read, Selective Write, Selective Forget - The Whiteboard Analogy
- Long Short Term Memory(LSTM) and Gated Recurrent Units(GRUs)
- How LSTMs avoid the problem of vanishing gradients
- How LSTMs avoid the problem of vanishing gradients (Contd.)
- Introduction to Encoder Decoder Models
- Applications of Encoder Decoder models
- Attention Mechanism
- Attention Mechanism (Contd.)
- Attention over images
- Hierarchical Attention