Lecture 23 of 70Information and statistical inference-1 (Unit 4)
Properties of KL divergence
Prof. Himanshu Tyagi · IISc Bangalore
Credits and licence
- From
- Information Theory, IISc Bangalore (at NPTEL)
- Licence
- CC BY-NC-SA, as NPTEL states it. Non-commercial use only.
- Attribution
- NPTEL states: "Distributed under Creative Commons Attribution-ShareAlike / CC BY - NC - SA", beside a BY-NC-SA badge.
- Who can watch
- Tweens profiles and older. How courses are placed
All the lectures
- What is information?
- How to model uncertainty?
- Basic concepts of probability
- Estimates of random variables
- Limit theorems
- Review
- Source model
- Motivating examples
- A compression problem
- Shannon entropy
- Random hash
- Review 2
- Uncertainty and randomness
- Total variation distance
- Generating almost random bits
- Generating samples from a distribution using uniform randomness
- Typical sets and entropy
- Review 3
- Hypothesis testing and estimation
- Examples.
- The log-likelihood ratio test
- Kullback-Leibler divergence and Stein's lemma
- Properties of KL divergence
- Review 4
- Information per coin-toss
- Multiple hypothesis testing
- Error analysis of multiple hypothesis testing
- Mutual information
- Fano's inequality
- Measures of information
- Chain rules
- Chain rules
- Shape of measures of information
- Data processing inequality
- Midyear Review
- Proof of Fano's inequality
- Capacity as information radius
- Proof of Pinsker's inequality
- Continuity of entropy
- Lower bound for compression
- Lower bound for hypothesis testing
- Review 7
- Lower bound for random number generation
- Strong converse
- Lower bound for minmax statistical estimation
- Variable length source codes
- Review 8
- Kraft's inequality
- Shannon code
- Huffman code
- Minmax Redundancy
- Type based universal compression
- Review 9
- Arithmetic code
- Online probability assignment
- Compression of databases: A scheme
- Compression of databases: A lower bound
- Repetition code
- Channel capacity
- Sphere packing bound for BSC
- Random coding bound for BSC
- Random coding bound for general channel
- Review 11
- Converse proof for channel coding theorem
- Additive Gaussian Noise channel
- Mutual information and differential entropy
- Channel coding theorem for Gaussan channel
- Parallel channels and water-filling
- Live Session 23-09-2020
- Live Session 07-10-2020