Learn · NPTEL

Course10810816870 lectures

Information Theory

Prof. Himanshu Tyagi · IISc Bangalore

Start with lecture 1 At NPTEL

Lectures

Information and probabilistic modeling

  1. What is information? Video
  2. How to model uncertainty? Video
  3. Basic concepts of probability Video
  4. Estimates of random variables Video
  5. Limit theorems Video

Uncertainty, compression, and entropy (Unit 2)

  1. Review Video
  2. Source model Video
  3. Motivating examples Video
  4. A compression problem Video
  5. Shannon entropy Video
  6. Random hash Video

Randomness and entropy (unit 3)

  1. Review 2 Video
  2. Uncertainty and randomness Video
  3. Total variation distance Video
  4. Generating almost random bits Video
  5. Generating samples from a distribution using uniform randomness Video
  6. Typical sets and entropy Video

Information and statistical inference-1 (Unit 4)

  1. Review 3 Video
  2. Hypothesis testing and estimation Video
  3. Examples. Video
  4. The log-likelihood ratio test Video
  5. Kullback-Leibler divergence and Stein's lemma Video
  6. Properties of KL divergence Video

Information and statistical inference-2 (Unit 5)

  1. Review 4 Video
  2. Information per coin-toss Video
  3. Multiple hypothesis testing Video
  4. Error analysis of multiple hypothesis testing Video
  5. Mutual information Video
  6. Fano's inequality Video

Properties of measures of Information-1 (Unit 6)

  1. Measures of information Video
  2. Chain rules Video
  3. Chain rules Video
  4. Shape of measures of information Video
  5. Data processing inequality Video

Properties of measures of Information-2 (Unit 7) and Information Theoretic lower bounds (Unit 8)

  1. Midyear Review Video
  2. Proof of Fano's inequality Video
  3. Capacity as information radius Video
  4. Proof of Pinsker's inequality Video
  5. Continuity of entropy Video
  6. Lower bound for compression Video
  7. Lower bound for hypothesis testing Video

Information theoretic lower bounds (Unit 8) and Data Compression-1 (Unit 9)

  1. Review 7 Video
  2. Lower bound for random number generation Video
  3. Strong converse Video
  4. Lower bound for minmax statistical estimation Video
  5. Variable length source codes Video

Data Compression-1 (Unit 9) and Data Compression-2 (Unit 10)

  1. Review 8 Video
  2. Kraft's inequality Video
  3. Shannon code Video
  4. Huffman code Video
  5. Minmax Redundancy Video
  6. Type based universal compression Video

Data Compression-2 (Unit 10) and Data Compression-3 (Unit 11)

  1. Review 9 Video
  2. Arithmetic code Video
  3. Online probability assignment Video
  4. Compression of databases: A scheme Video
  5. Compression of databases: A lower bound Video

Channel coding and capacity (Unit 12) and Shannon's channel coding theorem proof (Unit 13)

  1. Repetition code Video
  2. Channel capacity Video
  3. Sphere packing bound for BSC Video
  4. Random coding bound for BSC Video
  5. Random coding bound for general channel Video

Shannon's channel coding theorem proof (Unit 13) and Gaussian channels (Unit 14)

  1. Review 11 Video
  2. Converse proof for channel coding theorem Video
  3. Additive Gaussian Noise channel Video
  4. Mutual information and differential entropy Video
  5. Channel coding theorem for Gaussan channel Video
  6. Parallel channels and water-filling Video

Live Session

  1. Live Session 23-09-2020 Video
  2. Live Session 07-10-2020 Video

Syllabus and materials

At NPTEL, as the course publishes them.

Credits and licence

Institution
IISc Bangalore
Department
Electrical Engineering
Teacher
Prof. Himanshu Tyagi
Level
Undergraduate
Language
English
Subjects
Engineering
Source
NPTEL
Licence
CC BY-NC-SA, as NPTEL states it. Non-commercial use only. NPTEL's terms
Attribution
NPTEL states: "Distributed under Creative Commons Attribution-ShareAlike / CC BY - NC - SA", beside a BY-NC-SA badge.
Who can watch
NPTEL gives no rating. Invisible Network shows it to Tweens profiles and older, judged from what the source says about it. How courses are placed