Learn · Introduction to Information Retrieval
Lecture 41 of 71Week 7
Introducing Unigram Language Model
Prof. Dwaipayan Roy · IISER Kolkata
Credits and licence
- From
- Introduction to Information Retrieval, IISER Kolkata (at NPTEL)
- Licence
- CC BY-NC-SA, as NPTEL states it. Non-commercial use only.
- Attribution
- NPTEL states: "Distributed under Creative Commons Attribution-ShareAlike / CC BY - NC - SA", beside a BY-NC-SA badge. Coordinated by IIT Madras.
- Who can watch
- Tweens profiles and older. How courses are placed
All the lectures
- Introduction to the course
- Introduction to Information Retrieval
- Text Processing: Term Selection
- Text Processing: Term Filtering and Linguistic Processing
- Indexing
- Inverted Index
- Making of the Inverted Index
- Storing Inverted Index
- Inverted Index Summary
- Statistical Properties of Text - Zipf's Law
- Statistical Properties of Text - Heap's Law
- Index Compression
- Getting started with PyLucene
- Installing Docker and PyLucene
- Understanding Different Fields in PyLucene - 1
- Understanding Different Fields in PyLucene - 2
- Reading a Small Toy Dataset for Indexing using PyLucene
- Different Analyzers in PyLucene
- Indexing using PyLucene - Setting the Parameters
- Indexing using PyLucene - 2
- Non-English Text Analysis using PyLucene
- Exploring Luke: The Lucene Index Viewer
- Experimenting with Different Options in Indexing
- PyLucene Practice Programs - 1
- PyLucene Practice Programs - 2
- Query Processing
- Introduction to Ranked Retrieval
- Introduction to Term Weightings for Retrieval - TF
- Introduction to Term Weightings for Retrieval - IDF
- Vector Space Model - 1
- Vector Space Model - 2
- Vector Space Model - 3
- Vector Space Model - Problem Solving - 1
- Vector Space Model - Problem Solving - 2
- Probabilistic Model - Introduction
- Binary Independence Model (BIM)
- Best Match Model, BM1, BM11, BM15, BM25
- Dissecting BM25, BM25 vs VSM, BM25 for long queries, BM25F, BM25+
- BM25, After 30 years
- Introduction to Language Model for Information Retrieval
- Introducing Unigram Language Model
- Estimating Document Language Model
- Zero Frequency Problem and introduction to Smoothing
- Jelinek-Mercer and Dirichlet Smoothed Language Model
- Comparing Smoothing with IDF and Summary of LM based Retrieval
- Using Divergence Measures for Document Scoring
- PyLucene for Retrieval
- PyLucene - Varius Query Classes 1 - TermQuery
- PyLucene - Varius Query Classes 2 - PhraseQuery, TermRangeQuery, Numerical Range Query, PrefixQuery
- PyLucene - Varius Query Classes 3 - BooleanQuery, WildcardQuery, FuzzyQuery, MatchAllDocsQuery
- Evaluating Information Retrieval Systems 1 - Set Based Evaluation Metrics
- Evaluating Information Retrieval Systems 2 - Precision@K, R-Prec, Incorporating Ranking in Precision and Recall
- Evaluating Information Retrieval Systems 3 - AP, MAP, GMAP, MRR
- Evaluating Information Retrieval Systems 4 - Graded relevance, nDCG
- Evaluating Information Retrieval Systems 5 - Hypothesis Testing, Role of Evaluation Forums, Kappa measure
- Indexing in Benchmark Dataset - Indexing TREC-like Benchmark Datasets
- Retrieval in Benchmark Dataset - Retrieval in TREC-like Benchmark Datasets, and Evaluation using TREC_EVAL
- Comparing Retrieval Models - Hypothesis Testing in Information Retrieval
- Relevance Feedback 1 - An Introduction
- Relevance Feedback 2 - Rocchio's Algorithm
- Relevance Feedback 3 - Incorporating Pseudo Feedback using Relevance based Language Model, RM3
- Web Search And Crawler
- PageRank Algorithm
- PageRank - Random Surfer's Algorithm
- HITS Algorithm: Hubs and Authorities
- Search Engine Optimization
- Learning to Rank for Information Retrieval
- Latent Semantic Indexing
- Introduction to Embeddings for Information Retrieval
- From Embeddings to Transformers
- Large Language Models for Information Retrieval