Learn · Applied Natural Language Processing
Lecture 41 of 92Week 5
One word learning architecture
Prof. Ramaseshan R · Chennai Mathematical Institute
Credits and licence
- From
- Applied Natural Language Processing, Chennai Mathematical Institute (at NPTEL)
- Licence
- CC BY-NC-SA, as NPTEL states it. Non-commercial use only.
- Attribution
- NPTEL states: "Distributed under Creative Commons Attribution-ShareAlike / CC BY - NC - SA", beside a BY-NC-SA badge. Coordinated by IIT Madras.
- Who can watch
- Tweens profiles and older. How courses are placed
All the lectures
- Introduction
- Operations on a Corpus
- Probability and NLP
- Vector Space models
- Sequence Learning
- Machine Translation
- Preprocessing
- Statistical Properties of Words - Part 01
- Statistical Properties of Words - Part 02
- Statistical Properties of Words - Part 03
- Vector Space Models for NLP
- Document Similarity - Demo, Inverted index, Exercise
- Vector Representation of words
- Contextual understanding of text
- Co-occurence matrix, n-grams
- Collocations, Dense word Vectors
- SVD, Dimensionality reduction, Demo
- Query Processing
- Topic Modeling
- Examples for word prediction
- Introduction to Probability in the context of NLP
- Joint and conditional probabilities, independence with examples
- The definition of probabilistic language model
- Chain rule and Markov assumption
- Generative Models
- Bigram and Trigram Language models -peeking indide the model building
- Out of vocabulary words and curse of dimensionality
- Exercise
- Naive-Bayes, classification
- Machine learning, perceptron, linearly separable
- Linear Models for Claassification
- Biological Neural Network
- Perceptron
- Perceptron Learning
- Logical XOR
- Activation Functions
- Gradient Descent
- Feedforward and Backpropagation Neural Network
- Why Word2Vec?
- What are CBOW and Skip-Gram Models?
- One word learning architecture
- Forward pass for Word2Vec
- Matrix Operations Explained
- CBOW and Skip Gram Models
- Building Skip-gram model using Python
- Reduction of complexity - sub-sampling, negative sampling
- Binay tree, Hierarchical softmax
- Mapping the output layer to Softmax
- Updating the weights using hierarchical softmax
- Discussion on the results obtained from word2vec
- Recap and Introduction
- ANN as a LM and its limitations
- Sequence Learning and its applications
- Introuduction to Recurrent Neural Network
- Unrolled RNN
- RNN - Based Language Model
- BPTT - Forward Pass
- BPTT - Derivatives for W,V and U
- BPTT - Exploding and vanishing gradient
- LSTM
- Truncated BPTT
- GRU
- Introduction and Historical Approaches to Machine Translation
- What is SMT?
- Noisy Channel Model, Bayes Rule, Language Model
- Translation Model, Alignment Variables
- Alignments again!
- IBM Model 1
- IBM Model 2
- Introduction to Phrase-based translation
- Symmetrization of alignments
- Extraction of Phrases
- Learning/estimating the phrase probabilities using another Symmetrization example
- Introduction to evaluation of Machine Translation
- BLEU - "A short Discussion of the seminal paper"
- BLEU Demo using NLTK and other metrics
- Encoder-Decoder model for Neural Machine Translation
- RNN Based Machine Translation
- Recap and Connecting Bloom Taxonomy with Machine Learning
- Introduction to Attention based Translation
- Research Paper discussion on "Neural machine translation by jointly learning to align and translate"
- Typical NMT architecture architecture and models for multi-language translation
- Beam Search
- Variants of Gradient Descend
- Introduction to Conversation Modeling
- A few examples in Conversation Modeling
- Some ideas to Implement IR-based Conversation Modeling
- Discussion of some ideas in Question Answering
- Hyperspace Analogue to Language - HAL
- Correlated Occurence Analogue to Lexical Semantic - COALS
- Global Vectors - Glove
- Evaluation of Word vectors