Where this topic leads
Topics that build on Information Theory Foundations
Once you have Information Theory Foundations, these are the topics that cite it as a prerequisite. Pick by tier and the area you want to push into next.
Editor's suggested next (13)
- Variational Autoencoders
- Fano Inequality
- Maximum Likelihood Estimation: Theory, Information Identity, and Asymptotic Efficiency
- Bits, Nats, Perplexity, and BPB
- CLIP, OpenCLIP, and SigLIP: Contrastive Language-Image Pretraining
- Cross-Entropy Loss: MLE, KL Divergence, and Classification
- Information Bottleneck
- Kelly Criterion
- KL Divergence
- Perplexity and Language Model Evaluation
- Representation Learning Theory
- Token Prediction and Language Modeling
- Tokenization and Information Theory
Core flagship topics (5)
- CLIP, OpenCLIP, and SigLIP: Contrastive Language-Image Pretraininglayer 4 · beyond-llms
- Cross-Entropy Loss: MLE, KL Divergence, and Classificationlayer 1 · ml-methods
- KL Divergencelayer 1 · foundations
- Maximum Likelihood Estimation: Theory, Information Identity, and Asymptotic Efficiencylayer 0B · statistical-estimation
- Variational Autoencoderslayer 3 · ml-methods
Standard topics (6)
- Bits, Nats, Perplexity, and BPBlayer 3 · llm-construction
- Fano Inequalitylayer 3 · statistical-foundations
- Kelly Criterionlayer 2 · decision-theory
- Perplexity and Language Model Evaluationlayer 3 · llm-construction
- Representation Learning Theorylayer 3 · modern-generalization
- Token Prediction and Language Modelinglayer 3 · llm-construction
Advanced or specialty topics (2)
- Information Bottlenecklayer 3 · modern-generalization
- Tokenization and Information Theorylayer 4 · llm-construction